Why am I seeing duplicate messages in Amazon SQS for the same Amazon S3 event?
Last updated: 2019-12-17
I'm seeing duplicate messages in Amazon Simple Queue Service (Amazon SQS) for the same Amazon Simple Storage Service (Amazon S3) event. Why is this happening?
Amazon S3 is designed to deliver notifications with a high degree of reliability using built-in backoff and retry mechanisms. In rare occasions, the retry mechanism might cause duplicate notifications for the same object event.
Amazon S3 event notifications are delivered as JSON objects that contain a sequencer key, which is a hexadecimal value that can be used to identify the event sequence of PUTs and DELETEs for the same object. Duplicate event notifications for a specific object event have the same value for the sequencer key.
If your application requires that duplicate notifications are identified, it's a best practice to maintain a secondary database or index of S3 objects using event notifications. Then, store and compare the sequencer key values to check for duplicates as each event notification is processed.