How do I troubleshoot InvalidSequenceToken errors in the CloudWatch logs?
Last updated: 2022-03-24
I receive an InvalidSequenceToken error in the log files when I send logs to Amazon CloudWatch using the CloudWatch agent. How can I troubleshoot and resolve this error?
The CloudWatch agent uses the PutLogEvents API to publish to CloudWatch. This API uses a sequence token for every consecutive PutLogEvents call, and each log stream has a sequence token associated with it. When the CloudWatch agent makes the PutLogEvents call, a sequence token is provided (unless writing to a newly created log stream). In response, CloudWatch returns the next expected sequence token to that source. The returned sequence token needs to be used for the next pushed log event.
If CloudWatch receives an event with a sequence token that it isn't expecting for a log system, it returns the InvalidSequenceToken error with the sequence token it was expecting. If needed, you can use the DescribeLogStreams API call to get an updated token.
The InvalidSequenceToken error occurs when:
- You've recently installed or restarted the CloudWatch agent service
- Multiple sources are sending logs to the same streams
InvalidSquenceToken errors after the CloudWatch agent restarts
You might see this error after the CloudWatch agent restarts. When the CloudWatch agent service has just started or restarted, the first attempt to push the logs fails. This is expected behavior that happens because the log stream already exists in the CloudWatch logs. As a result, the CloudWatch agent doesn't know which sequence token to use. Because it has a built-in retry mechanism, the CloudWatch agent picks up the right sequence token and uses it to push the logs again.
InvalidSequenceToken errors when multiple sources send logs to the same log stream
You receive an InvalidSequenceToken error if you have multiple instances sending logs to the same log stream. Similarly, if you have multiple log files in the same instance sending logs to the same log stream, you receive an error. This happens because the sequence token that was sent in the PutLogEvents call does not match the token the stream was expecting.
Resolve InvalidSequenceToken errors
Review the agent configuration files and be sure that you're not sending logs from multiple sources to the same log stream. Configuring multiple log sources to send data to a single log steam is not supported. Update the agent configuration file to use unique log stream names for each source. Then, restart the agent service using the fetch-config command.