I'm having trouble pushing log data to Amazon CloudWatch Logs using the CloudWatch Logs Agent (awslogs). How do I troubleshoot this problem?

Before you begin, confirm that the awslogs agent is able to connect to the CloudWatch Logs API endpoint.

Be sure that your configuration has:

  • Internet connectivity
  • Valid security group configurations
  • Valid network access control lists (network ACLs)

Fingerprinting issues

Review the header lines of the source log file. You set this file's path when configuring the data to be pushed to CloudWatch.

  • If the first few lines are blank or contain non-event data that stays the same, there might be issues with the log-identifying hash.
  • If the header lines are the same, then update the file_fingerprint_lines option in the agent configuration file. Be sure to specify which lines in each file are used for generating the identifying hash.

Check the awslogs log file for errors

Review the /var/log/awslogs.log log file. Be sure to note any error messages.

Permissions errors include:

  • NoCredentialsError: Unable to locate credentials If you haven't added an AWS Identity and Access Management (IAM) role to the instance, create and attach an IAM role. If you've already added an IAM role to the instance, update the IAM user credentials in the /etc/awslogs/awscli.conf file.
  • ClientError: An error occurred (AccessDeniedException) when calling the PutLogEvents operation: User: arn:aws:iam::012345678910:<role/user>/<iam-user-name> is not authorized to perform: logs:PutLogEvents[...] Configure the IAM role or user with the required permissions for CloudWatch Logs.

Timestamp errors include:

  • Fall back to previous event time: {'timestamp': 1492395793000, 'start_position': 17280L, 'end_position': 17389L}, previousEventTime: 1492395793000, reason: timestamp could not be parsed from message. Confirm that the log events begin with a timestamp. Check if the datetime_format specified in /etc/awslogs/awslogs.conf matches the timestamp format of the log events. Change the datetime_format to match the timestamp format as needed.
  • No file is found with given path '<PATH-TO-FILE>' Update the log file path in the agent configuration file to the correct path.
  • Caught exception: An error occurred (InvalidSequenceTokenException) when calling the PutLogEvents operation: The given sequenceToken is invalid[…] -or- Multiple agents might be sending log events to log stream[…] You can't push logs from multiple log files to a single log stream. Update your configuration to push each log to a log stream-log group combination.

Other awslogs issues

  • If logs stopped pushing after a log rotation, check the supported log rotation methods. For more information, see CloudWatch Logs Agent FAQs.
  • If logs are only pushed for a short time after the awslogs agent is restarted, check for duplicates in the [logstream] section of the agent configuration file. Each section must have a unique name.
  • If the awslogs.log log file takes up too much disk space, check the log file for errors and correct them. If the log file only contains informational messages, specify a lower logging level for the logging_config_file option in the agent configuration file.

Further troubleshooting

For further troubleshooting, note the instance-id (your instance's ID) collect and review the following information based on your configuration.

Yum installations:

  • yum version
$ yum info awslogs
$ yum info aws-cli-plugin-cloudwatch-logs
  • /etc/awslogs/awslogs.conf file
  • /etc/awslogs/awscli.conf file
  • Other relevant files in /etc/awslogs/
  • /var/log/awslogs.log file

Script-based installations:

  • The awslogs version, obtained with the following command:
$ /var/awslogs/bin/awslogs-version.sh
  • /var/awslogs/etc/awslogs.conf file
  • /var/awslogs/etc/awscli.conf file
  • Other relevant files in /var/awslogs/etc/
  • /var/log/awslogs.log
  • /var/log/awslogs-agent-setup.log

For rotation-related issues, also collect and review:

  • A snippet of the source logs.
  • A list of the monitoring target directory's contents. Use the command ls -la with the directory path to obtain this:
$ ls -la <Monitoring-Target-Directory-Path>

Did this page help you? Yes | No

Back to the AWS Support Knowledge Center

Need help? Visit the AWS Support Center.

Published: 2018-12-06