Why do I see "audit: backlog limit exceeded" errors in my EC2 Linux instance's screenshot and system logs, and what can I do to avoid this?

Last updated: 2020-09-18

I see "audit callbacks suppressed" and "audit: backlog limit exceeded" error messages in my Amazon Elastic Compute Cloud (Amazon EC2) Linux instance's screenshot and system logs. Why am I receiving these messages, and how can I prevent them from reoccurring?

Short description

The audit backlog buffer in a Linux system is a kernel level socket buffer queue that the operating system uses to maintain or log audit events. When a new audit event triggers, the system logs the event and adds it to the audit backlog buffer queue.

The backlog_limit parameter value is the number of audit backlog buffers. The parameter is set to 320 by default, as shown in the following example:

# auditctl -s
enabled 1
failure 1
pid 2264
rate_limit 0
backlog_limit 320
lost 0
backlog 0

Audit events logged beyond the default number of 320 cause the following errors on the instance:

audit: audit_backlog=321 > audit_backlog_limit=320 

audit: audit_lost=44393 audit_rate_limit=0 audit_backlog_limit=320 

audit: backlog limit exceeded

-or-

audit_printk_skb: 153 callbacks suppressed

audit_printk_skb: 114 callbacks suppressed

An audit buffer queue at or exceeding capacity might also cause the instance to hang or remain in an unresponsive state.

To avoid backlog limit exceeded errors, increase the backlog_limit parameter value. Large servers have a larger number of audit logs triggered, so increasing buffer space helps avoid error messages.

Note: Increasing the audit buffer consumes more of the instance's memory. How large you make the backlog_limit parameter depends on the total memory of the instance. If the system has enough memory, you can try doubling the existing backlog_limit parameter value.

The following is a calculation of the memory required for the auditd backlog. Use this calculation to determine how large you can make the backlog queue without causing memory stress on your instance.

One audit buffer = 8970 Bytes
Default number of audit buffers (backlog_limit parameter) = 320
320 * 8970 = 2870400 Bytes, or 2.7 MiB

The size of the audit buffer is defined by the MAX_AUDIT_MESSAGE_LENGTH parameter. For more information, see MAX_AUDIT_MESSAGE_LENGTH in the Linux audit library on github.com.

Note: If your instance is inaccessible and you see backlog limit exceeded messages in the system log, stop and start the instance. Then, perform the following steps to change the audit buffer value.

Resolution

Note: In this example, we're changing the backlog_limit parameter value to 8192 buffers. 8192 buffers equals 70 MiB of memory based on the preceding calculation. You can use any value based on your memory calculation.

1. Access the instance using SSH.

2. Verify the current audit buffer size.

Note: The backlog_limit parameter is listed as -b. For more information, see auditctl(8) on the auditctl-man-page

Amazon Linux 1 and other operating systems that don't have systemd:

$ sudo cat /etc/audit/audit.rules
# This file contains the auditctl rules that are loaded
# whenever the audit daemon is started via the initscripts.
# The rules are simply the parameters that would be passed
# to auditctl.

# First rule - delete all
-D

# Increase the buffers to survive stress events.
# Make this bigger for busy systems
-b 320

# Disable system call auditing.
# Remove the following line if you need the auditing.
-a never,task

# Feel free to add below this line. See auditctl man page

Amazon Linux 2 and other operating systems that use systemd:

$ sudo cat /etc/audit/audit.rules
# This file is automatically generated from /etc/audit/rules.d
-D
-b 320
-f 1

3. Access the audit.rules file using an editor, such as the vi editor:

Amazon Linux 1 and other operating systems that don't use systemd:

$ sudo vi /etc/audit/audit.rules

Amazon Linux 2 and other operating systems that use systemd:

$ sudo vi /etc/audit/rules.d/audit.rules

4. Edit the -b parameter to a larger value. The following example changes the -b value to 8192.

$ sudo cat /etc/audit/audit.rules
# This file contains the auditctl rules that are loaded
# whenever the audit daemon is started via the initscripts.
# The rules are simply the parameters that would be passed
# to auditctl.

# First rule - delete all
-D

# Increase the buffers to survive stress events.
# Make this bigger for busy systems
-b 8192

# Disable system call auditing.
# Remove the following line if you need the auditing.
-a never,task

# Feel free to add below this line. See auditctl man page

$ sudo auditctl -s
enabled 1
failure 1
pid 2264
rate_limit 0
backlog_limit 320
lost 0
backlog 0

5. Restart the auditd service. The new backlog_limit value takes effect. The value also updates in auditctl -s, as shown in the following example:

# sudo service auditd stop
Stopping auditd:                                           [  OK  ]
# sudo service auditd start
Starting auditd:                                           [  OK  ]
# auditctl -s
enabled 1
failure 1
pid 26823
rate_limit 0
backlog_limit 8192
lost 0
backlog 0

Note: If your instance is inaccessible and you see backlog limit exceeded messages in the system log, stop and start the instance. Then, perform the preceding steps to change the audit buffer value.


Did this article help?


Do you need billing or technical support?