March 6, 2015

We have detected that certain Amazon EC2 instances running Windows Server 2012 R2 lose network and storage connectivity after an instance reboot. This applies to Windows Server 2012 R2 Amazon Machine Images (AMI) made available before September 10th, 2014.

The loss of network connectivity is triggered by a feature in Windows Server 2012 R2 (Plug and Play Cleanup) that scans for and disables inactive system devices every 30 days. Windows is identifying the EC2 network device as inactive and removing it from the system. If this happens, the instance will lose network connectivity after any subsequent reboot.

For Windows Server 2012 R2 AMIs made available before September 10th, 2014, we are providing an in-place driver upgrade. You can download the software package and run it on your existing instances.

Microsoft Windows Server 2012 R2 AMIs published between September 10th and October 14th, 2014 are pre-configured with a disabled Plug and Play Cleanup task, so you can stop and start, resize and reboot as needed.
Windows Server 2012 R2 AMIs published on October 15th, 2014 and newer include an updated paravirtualization (PV) driver that is compatible with the Plug and Play Cleanup task.

If you are unable to perform the in-place driver upgrade on your pre-September 10th, 2014 instance, we are providing a script to help. The script will allow you to quickly determine if your instance is affected. If it is affected, and the device has not yet been removed, the script will disable the Plug and Play Cleanup scan. If the device has already been removed, the script will repair the device, disable the Plug and Play Cleanup scan, and allow your instance to reboot with network connectivity enabled.


Q: Why does the Plug and Play Cleanup scan remove my device?

The Plug and Play Cleanup scan identifies an AWS device (the EC2 network device) as inactive and removes it, causing network inaccessibility and disk access issues after reboot.

Q: What has AWS done to help fix this?

Initially, we provided a remediation script to help and advised you to migrate to the latest Microsoft Windows Server 2012 R2 AMI, which came with Plug and Play Cleanup disabled by default. Then we released Windows Server 2012 R2 AMI with updated drivers. Finally, we released an in-place driver upgrade for existing instances.

Q: How do I find the latest Windows Server 2012 R2 AMI?
The latest Windows Server 2012 R2 AMIs can be selected from the “Quick Start” menu, when you select “Launch Instance” from the AWS Management Console.

Q: What is the change to the latest Windows Server 2012 R2 AMI that addresses this issue?

The latest Windows Server 2012 R2 AMIs come with updated drivers that are compatible with the Plug and Play Cleanup task.

Q: How do I determine which driver I am using?

To determine which driver is installed on a Windows Server 2012 R2 instance, you can check the version number within device manager.

Q: What action should I take given the different Windows Server 2012 R2 driver versions?

  • For AMIs published before September 10th, 2014, the AWS PV driver will be listed as v.7.2.2. We recommend you perform an in-place driver upgrade or update to an AMI published on October 15th 2014 or later.
  • For AMIs published between September 10th and October 14th, 2014, the AWS PV driver will be listed as v.7.2.2.1. These AMIs are pre-configured with the Plug and Play Cleanup task disabled, so you can stop and start, resize and reboot your instance as needed. We recommend that you perform an in-place driver upgrade or update to an AMI published on October 15th, 2014 or later. These AMIs include a PV driver that is compatible with the Plug and Play Cleanup task.
  • For AMIs published on October 15th, 2014 and later, the AWS PV driver will be listed as v.7.2.4 or beyond. No further action is required.

Q: Is an in-place driver upgrade available?

Yes. We released a software package that allows you to upgrade the drivers on your existing instances.

Q: When should I use the in-place driver upgrade?

If you are unable to migrate to an instance that was published on October 15, 2014 or later, we recommend you perform an in-place driver upgrade.

Q: How do I perform the in-place driver upgrade?

To get started, download the latest software package and refer to the README file.

Q: Will the in-place driver upgrade require me to reboot my instance?

Yes. The upgrade will require you to reboot your instance. You should follow change management best practices, such as making back-ups, when you perform software updates to your instances.

Q: What if I am unable to perform the in-place driver upgrade or migrate to a newer instance?

You should run the remediation script.

Q: What does the script do?

The script determines if your instance is affected and is designed to disable the Plug and Play Cleanup scan that runs every 30 days. The script also repairs the device allowing your instance to reboot with network connectivity enabled.

Q: Can I automate the execution of the script?

Yes. You can use Microsoft System Center and Group Policy, as well as other automation and distribution tools to execute the script broadly.

Q: What can I expect after I run the script?

The script reports whether or not Plug and Play Cleanup affected an instance. If your instance is affected, the script disables Plug and Play Cleanup and restores the device. You can reboot your instance, but you will be unable to stop, then start or resize your instance.

Q: Why can’t I stop, then start or resize my instance after running the script?

An updated driver is needed in order to perform a full stop, then start or resize.

Q: I don’t have a support contract. Will AWS Support help me?

Any customer running an instance failing health checks can open a complementary case with Support via Support for Health Checks.

Q: Is it possible to confirm whether an inaccessible Windows Server 2012 R2 instance is affected by the Plug and Play cleanup task?

Yes. When viewing the system log output of the instance in the AWS management console, you will see the message: “Difficulty detecting PV driver details for Console Output.”

You can remediate this issue by following the steps below:

  1. Stop the instance
  2. Create a backup (Right click the instance in the AWS Management Console, Select Image, Create Image)
  3. Change the instance type to C3 (any c3)
  4. Start the instance
  5. RDP into the instance
  6. Download the latest AWS PV upgrader 
  7. Unzip and run AWSPVDriverSetup.msi
  8. The OS will reboot automatically after running the upgrader, and install the driver. During this time the instance will be unavailable (5-15 minutes). Once complete, the instance will again be accessible.
  9. Verify driver has been installed by checking device manager and under storage controllers look for AWS PV Storage Host Adapter and check the driver properties and make sure the version is correct
  10. Stop the instance
  11. Change instance type back to the original instance type
  12. Start the instance, login, and use as needed