EC2 Maintenance Update II

I’d like to give you an update on the EC2 Maintenance announcement that I posted last week. Late yesterday (September 30^th), we completed a reboot of less than 10% of the EC2 fleet to protect you from any security risks associated with the Xen Security Advisory (XSA-108).

This Xen Security Advisory was embargoed until a few minutes ago; we were obligated to keep all information about the issue confidential until it was published. The Xen community (in which we are active participants) has designed a two-stage disclosure process that operates as follows:

• Early disclosure to select organizations (a list maintained and regularly evaluated by the Xen Security Team based on a set of public criteria established by the Xen Project community) with a limited time to make accommodations and apply updates before it becomes widely known.
• Full disclosure to everyone on the public disclosure date.

Because our customers security is our top priority and because the issue was potentially harmful to our customers, we needed to take fast action to protect them. For the reasons mentioned above, we couldnt be as expansive as wed have liked on why we had to take such fast action.

The zone by zone reboots were completed as planned and we worked very closely with our customers to ensure that the reboots went smoothly for them.

We’ll continue to be vigilant and will do our best to protect all AWS customers from similar issues in the future. As an AWS user, you may also want to take this opportunity to re-examine your AWS architecture to look for possible ways to make it even more fault-tolerant. Here are a few suggestions to get you started:

• Run instances in two or more Availability Zones.
• Pay attention to your Inbox and to the alerts on the AWS Management Console. Make sure that you fill in the “Alternate Contacts” in the AWS Billing Console.
• Review the personalized assessment of your architecture in the Trusted Advisor, then open up AWS Support Cases to get engineering assistance as you implement architectural best practices.
• Use Chaos Monkey to induce various kinds of failures in a controlled environment.
• Examine and consider expanding your use of Amazon Route 53 and Elastic Load Balancing checks to ensure that web traffic is routed to healthy instances.
• Use Auto Scaling to keep a defined number of healthy instances up and running.

You should also consult our Overview of Security Practices whitepaper for more information around AWS and security.

— Jeff;