ELB Connection Draining – Remove Instances From Service With Care

AWS Elastic Load Balancing distributes incoming traffic across multiple Amazon EC2 instances:

You can use Elastic Load Balancing on its own, or in conjunction with Auto Scaling. When combined, the two features allow you to create a system that automatically adds and removes EC2 instances in response to changing load:

In either case, the Elastic Load Balancer equitably routes traffic across all of the instances that are currently registered with it and deemed to be healthy based on configured health checks

In a large, active system, several different types of application and capacity management activities will take place from time to time. For example:

• Auto Scaling will add a new instance to the Auto Scaling Group.
• Auto Scaling will remove an existing instance from the Auto Scaling Group.
• An existing instance will fail its health checks and be taken out of service.
• Running software on an existing instance will be updated in-place.
• Existing instances will be gradually replaced with new ones that contain updated software.

While these activities are potentially underway, the Elastic Load Balancer is accepting incoming requests and routing them to healthy instances. A busy system might be processing tens of thousands of requests per second. Some of these requests might finish within milliseconds. Others, perhaps backed by complex database queries or involving file downloads, might take seconds or even tens of seconds.

Connection Draining
Today’s new feature sits squarely at the intersection of the activities I described above and the stream of incoming requests.

In order to provide a first-class use experience, you’d like to avoid breaking open network connections while taking an instance out of service, updating its software, or replacing it with a fresh instance that contains updated software. Imagine each broken connection as a half-drawn web page, an aborted file download, or a failed web service call, each of which results in an unhappy user or customer.

You can now avoid this situation by enabling the new Connection Draining feature for your Elastic Load Balancers. You can do this from the AWS Management Console, the AWS Command Line Interface, or by calling the ModifyLoadBalancerAttributes function in the Elastic Load Balancing API. You simply enable the feature and enter a timeout between one second and one hour. Connection Draining is enabled by default for load balancers that are created using the Console.

In order to enable Connection Draining using the AWS Management Console you must use the new version of the EC2 console. To enable it, visit the EC2 tab, click on Load Balancers and look for the “cartoon bubble” in the top right corner:

Click on the bubble to enable the new version of the console;

Select one of your load balancers, and click on the Instances tab:

Look for Connection Draining and click on Edit:

Click to enable Connection Draining and set the desired timeout. Click Save and you are good to go!

When Connection Draining is enabled and configured, the process of deregistering an instance from an Elastic Load Balancer gains an additional step. For the duration of the configured timeout, the load balancer will allow existing, in-flight requests made to an instance to complete, but it will not send any new requests to the instance. During this time, the API will report the status of the instance as InService, along with a message stating that “Instance deregistration currently in progress.” Once the timeout is reached, any remaining connections will be forcibly closed.

AWS CloudFormation supports Connection Draining as well as the recently released Access Logs feature. Here is a sample AWS CloudFormation template (launch stack) for provisioning an Auto Scaling Group, load balanced with an Elastic Load Balancer with Connection Draining enabled, and Access Logs delivered to an S3 bucket.

You can start using this new feature today. Read the documentation for Connection Draining if you would like to learn more.

— Jeff;