How do I troubleshoot logging issues in Elastic Beanstalk?

Last updated: 2019-10-29

My log file is empty, or my log stream suddenly stopped streaming log messages. How can I resolve this issue in AWS Elastic Beanstalk?

Resolution

Choose one of the following options based on the logging issue that you're experiencing:

Find your empty log files

By default, Elastic Beanstalk keeps all logs on the Amazon Elastic Compute Cloud (Amazon EC2) instances in your environment, and periodically rotates your logs to manage file size and conserve disk space. Elastic Beanstalk exports your logs to Amazon Simple Storage Service (Amazon S3). For more information, see Log Rotation Settings on Linux.

To access recently rotated log messages, check /var/log/rotated.

Find out why your log file and log stream stopped outputting new log messages

Use Amazon CloudWatch Logs to stream your log files continuously from your Elastic Beanstalk EC2 instances.

If log streaming stops working after the first log rotation occurs on your EC2 instance, then your application might not be logging unique lines. For example, your application log could look as follows:

Hello my application!
- at time 1:00 PM
Hello my application!
- at time 2:00 PM
...

In the preceding example, not every line is unique. A unique entry spans two lines. It would take a minimum of two lines to uniquely identify this log file.

Note: The CloudWatch Logs agent identifies your log file and uses the unique number of lines that your file contains to detect when log rotation occurs.

To get log streaming to work again, try either of the following:

Edit the application log format to include a unique identifier (such as a timestamp) at the start of each line.

-or-

Edit the file_fingerprint_lines entry to count the lines of your file, until enough lines are included to uniquely identify the file.

Note: To edit this setting, you must customize the CloudWatch Logs agent configuration. The Elastic Beanstalk integration with CloudWatch Logs doesn't directly support the streaming of custom log files that your application generates. To stream custom logs, use a configuration file to directly install the CloudWatch Logs agent and to configure the files to be pushed.

For example, the following .ebextension customizes file_fingerprint_lines for Node.js application log files:

files:
  "/etc/awslogs/config/beanstalklogs.conf":
    mode: "000644"
    user: root
    group: root
    content: |
      [/var/log/nodejs/nodejs.log]
      log_group_name=/aws/elasticbeanstalk/streamlogs/var/log/nodejs/nodejs.log
      log_stream_name={instance_id}
      file=/var/log/nodejs/nodejs.log*
      file_fingerprint_lines=2-5
 
      [/var/log/nginx/error.log]
      log_group_name=/aws/elasticbeanstalk/streamlogs/var/log/nginx/error.log
      log_stream_name={instance_id}
      file=/var/log/nginx/error.log*
 
      [/var/log/nginx/access.log]
      log_group_name=/aws/elasticbeanstalk/streamlogs/var/log/nginx/access.log
      log_stream_name={instance_id}
      file=/var/log/nginx/access.log*
 
      [/var/log/httpd/error.log]
      log_group_name=/aws/elasticbeanstalk/streamlogs/var/log/httpd/error.log
      log_stream_name={instance_id}
      file=/var/log/httpd/error.log*
 
      [/var/log/httpd/access.log]
      log_group_name=/aws/elasticbeanstalk/streamlogs/var/log/httpd/access.log
      log_stream_name={instance_id}
      file=/var/log/httpd/access.log*
 
      [/var/log/eb-activity.log]
      log_group_name=/aws/elasticbeanstalk/streamlogs/var/log/eb-activity.log
      log_stream_name={instance_id}
      file=/var/log/eb-activity.log*
 
commands:
  01_remove_eb_stream_config:
    command: 'rm -fr /etc/awslogs/config/beanstalklogs.conf.bak'
  02_restart_log_agent:
    command: 'service awslogs restart'

Troubleshoot the failed creation of a log stream

If you enabled log streaming, but your log stream still isn't showing up, see Troubleshooting CloudWatch Logs Integration.