Why can’t I push log data to CloudWatch Logs with the awslogs agent?

Last updated: 2020-11-06

I'm unable to push log data to Amazon CloudWatch Logs using the CloudWatch Logs Agent (awslogs). How do I troubleshoot this?

Resolution

Before you begin, confirm that the awslogs agent can connect to the CloudWatch Logs API endpoint.

Be sure that your configuration has the following:

  • Internet connectivity
  • Valid security group configurations
  • Valid network access control lists (network ACLs)

Fingerprinting issues

Review the header lines of the source log file. You set this file's path when configuring the data to be pushed to CloudWatch.

  • If the first few lines are blank or contain non-event data that stays the same, there might be issues with the log-identifying hash.
  • If the header lines are the same, update the file_fingerprint_lines option in the agent configuration file. Be sure to specify what lines in each file are used for generating the identifying hash.

Check the awslogs log file for errors

Review the /var/log/awslogs.log log file. Be sure to note any error messages.

Permissions errors include:

  • NoCredentialsError: Unable to locate credentials – If you didn't add an AWS Identity and Access Management (IAM) role to the instance, create and attach an IAM role. If you already added an IAM role to the instance, update the IAM user credentials in the /etc/awslogs/awscli.conf file.
  • ClientError: An error occurred (AccessDeniedException) when calling the PutLogEvents operation: User: arn:aws:iam::012345678910:<role/user>/<iam-user-name> is not authorized to perform: logs:PutLogEvents[...]Configure the IAM role or user with the required permissions for CloudWatch Logs.

Timestamp errors include:

  • Fall back to previous event time: {'timestamp': 1492395793000, 'start_position': 17280L, 'end_position': 17389L}, previousEventTime: 1492395793000, reason: timestamp could not be parsed from message. – Confirm that the log events begin with a timestamp. Check if the datetime_format specified in /etc/awslogs/awslogs.conf matches the timestamp format of the log events. Change the datetime_format to match the timestamp format as needed.
  • No file is found with given path '<PATH-TO-FILE>' – Update the log file path in the agent configuration file to the correct path.
  • Caught exception: An error occurred (InvalidSequenceTokenException) when calling the PutLogEvents operation: The given sequenceToken is invalid[…] -or- Multiple agents might be sending log events to log stream[…] – You can't push logs from multiple log files to a single log stream. Update your configuration to push each log to a log stream-log group combination.

Other awslogs issues

  • If logs stopped pushing after a log rotation, check the supported log rotation methods. For more information, see CloudWatch Logs Agent FAQs.
  • If logs are pushed briefly only after the awslogs agent is restarted, check for duplicates in the [logstream] section of the agent configuration file. Each section must have a unique name.
  • If the awslogs.log log file takes up too much disk space, check the log file for errors and then correct them. If the log file contains only informational messages, specify a lower logging level for the logging_config_file option in the agent configuration file.

Further troubleshooting

For further troubleshooting, note the instance-id (your instance's ID). Then, collect and review the following based on your configuration.

Yum installations:

  • yum version
$ yum info awslogs
$ yum info aws-cli-plugin-cloudwatch-logs
  • /etc/awslogs/awslogs.conf file
  • /etc/awslogs/awscli.conf file
  • Other relevant files in /etc/awslogs/
  • /var/log/awslogs.log file

Script-based installations:

  • The awslogs version, obtained with the following command:
$ /var/awslogs/bin/awslogs-version.sh
  • /var/awslogs/etc/awslogs.conf file
  • /var/awslogs/etc/awscli.conf file
  • Other relevant files in /var/awslogs/etc/
  • /var/log/awslogs.log
  • /var/log/awslogs-agent-setup.log

For rotation-related issues, collect and review:

  • A snippet of the source logs
  • A list of the monitoring target directory's contents. Use the command ls -la with the directory path to obtain this:
$ ls -la <Monitoring-Target-Directory-Path>

Did this article help?


Do you need billing or technical support?