Why is my Amazon Elastic Compute Cloud (Amazon EC2) instance exceeding its network limits when average utilization is low?

Last updated: 2022-05-26

Why is my Amazon Elastic Compute Cloud (Amazon EC2) instance exceeding its network limits when average utilization is low?

Short description

You can query network performance metrics in real time on instances that support enhanced networking through the Elastic Network Adapter (ENA). These metrics provide a cumulative number of packets queued or dropped on each network interface since the last driver reset. The following are some of the ENA metrics:

  • bw_in_allowance_exceeded: The number of packets queued or dropped because the inbound aggregate bandwidth exceeded the maximum for the instance.
  • bw_out_allowance_exceeded: The number of packets queued or dropped because the outbound aggregate bandwidth exceeded the maximum for the instance.
  • pps_allowance_exceeded: The number of packets queued or dropped because the bidirectional Packets Per Second (PPS) exceeded the maximum for the instance.

In some cases, you might see queuing or drops even though your average bandwidth or PPS as seen in Amazon CloudWatch is low. For example, the NetworkIn, NetworkOut, NetworkPacketsIn, or NetworkPacketsOut metrics in CloudWatch might show amounts that don't suggest a limit being reached.

Resolution

Microbursts are the most common cause of the preceding symptoms. Microbursts are short spikes in demand followed by periods of low or no activity. These bursts only last for seconds, milliseconds, or even microseconds. In the case of microbursts, the CloudWatch metrics listed in the previous section aren't granular enough to reflect them.

How averages are calculated

The EC2 metrics in CloudWatch listed in the previous section are sampled every 1 minute. These metrics capture the total bytes or packets transferred in that period. These samples are then aggregated and published to CloudWatch in 5-minute periods. Each statistic in the period returns a different sample value:

  • Minimum is the sample value with the lowest byte / packet count.
  • Maximum is the sample value with the highest byte / packet count.
  • Sum is the sum of all sample values.
  • SampleCount is the number of aggregated samples (in this case, 5).
  • Average is the average sample value, calculated by dividing Sum by SampleCount.

Average throughput or PPS can be calculated in two ways:

  • Divide Sum by Period (e.g., 300 seconds), for a simple 5-minute average.
  • Divide Maximum by 60 seconds, for an average in the minute with the highest activity.

How microbursts are reflected in CloudWatch metrics

The following is an example of microburst, and how it is reflected in CloudWatch:

  • The instance has a network bandwidth performance of 10 Gbps (1.25 GB/s).
  • In a given sample (60 seconds), an outbound data transfer of 20 GB uses up all available bandwidth, causing bw_out_allowance_exceeded to increment. Transfer completes in about 20 seconds, and no further data is sent afterwards.
  • Instance remains idle for the remaining 4 samples (240 seconds).

In this example, the average throughput in a 5-minute period is much lower than the one during the microburst:

SUM(NetworkOut) / PERIOD = ((20 GB * 1 sample) + (0 GB * 4 samples)) / 300 seconds = ~0.066 GB/s * 8 bits = ~0.533 Gbps

Even if you calculate the throughput based on the highest sample, the average still doesn't reflect the throughput amount:

MAXIMUM(NetworkOut) / 60 seconds = 20 GB / 60 = ~0.333 GB/s * 8 bits = ~2.667 Gbps

Monitoring microbursts

To measure throughput and PPS at a more granular level, use operating system (OS) tools to monitor network statistics. Monitor your network statistics more frequently during periods of shaping or high activity.

The following are examples of operating system tools:

Linux

  • sar
  • nload
  • iftop
  • iptraf-ng

Windows

  • Performance Monitor

The CloudWatch agent can be used on both Linux and Windows to publish these OS-level network metrics to CloudWatch as custom metrics. These metrics can be published at intervals as low as 1 second. Be aware that high-resolution metrics (those with a period lower than 60 seconds) lead to higher charges. For more information about CloudWatch pricing, see Amazon CloudWatch pricing.

It's a best practice for you to monitor the network performance metrics provided by ENA. The driver version must be greater than or equal to 2.2.10 (Linux) and 2.2.2 (Windows) for you to be able to view the metrics. For more information, check the relevant pages for Linux and Windows. CloudWatch Agent can also publish ENA metrics.

For instructions on publishing ENA metrics for Linux, see Collect network performance metrics. For Windows, ENA metrics are available in Performance Monitor, and the CloudWatch Agent can publish any metrics available there.

Preventing microbursts

To avoid microbursts, traffic needs to be paced at the sender(s) so that it doesn't exceed a maximum throughput or packet rate. This makes microbursts difficult to avoid. Pacing traffic at the sender usually requires application-level changes. Depending on how this change is implemented, OS-support for traffic pacing might need to be available and turned on at the sender. That might not always be possible, or practical. Microbursting also happens as a result of too many connections sending packets in a short period, in which case pacing individual senders doesn't help.

Due to these reasons, it's a best practice to monitor ENA metrics, as stated previously. If limits are being reached often, or if there's evidence that this is impacting your application(s), do the following:

  • Scale up: Move to a larger instance size. Larger instances generally have higher allowances. Network optimized instances (such as instance with an "n", as in C5n), have higher allowances than their respective non network-optimized counterparts.
  • Scale out: Spread traffic across multiple instances to reduce traffic and contention at individual instances.

On Linux, in addition to the preceding options, there are potential mitigation options for advanced users:

  • SO_MAX_PACING_RATE: This socket option can be passed to setsockopt by an application to specify a maximum pacing rate (bytes per second). The Linux kernel then paces traffic from that socket so that it doesn't exceed the limit. This option requires the following:
    Application-level code changes.
    Support from the kernel.
    The use of Fair Queue (FQ) queuing discipline or the kernel's support for pacing at the TCP layer (applicable to TCP only).
  • Queuing disciplines (qdiscs): Some qdiscs provide support for traffic shaping at various levels. For more information, see the Traffic Control (TC) manual page.
  • Shallow Transmission (Tx) queues: In some scenarios, shallow Tx queues help reduce PPS throttling. It can be achieved in two ways:
    Byte Queue Limits (BQL): BQL dynamically limits the amount of in-flight bytes on Tx queues. BLQ is turned on by default on ENA driver versions shipped with the Linux kernel (those ending with a "K"). For ENA driver versions from GitHub (those ending with a "g"), BQL is available as of v2.6.1g, and is turned off by default. You can turn on BQL using the enable_bql ENA module parameter.
    Reduced Tx queue length: Reduce Tx queue length from its default of 1,024 packets to a lower amount (minimum 256). You can change this length using the ethtool.