Why is my EC2 Linux instance not booting and going into emergency mode?

Last updated: 2020-04-22

When I boot my Amazon Elastic Compute Cloud (Amazon EC2) Linux instance, the instance goes into emergency mode and the boot process fails. Then, the instance is inaccessible. How can I fix this?

Short Description

The most common reasons an instance might boot in emergency mode are:

  • A corrupted kernel.
  • Auto-mount failures because of incorrect entries in the /etc/fstab.

To verify what type of error is occurring, view the instance's console output. You might see a Kernel panic error message in the console output if the kernel is corrupted. Dependency failed messages appear in the console output if auto-mount failures occur.

Resolution

Kernel panic errors

Kernel panic error messages occur when the grub configuration or initramfs file is corrupted. If a problem with the kernel exists, you might see the error "Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(8,1)" in the console output.

To resolve kernel panic errors:

1.    Revert the kernel to a previous, stable kernel. For instructions on how to revert to a previous kernel, see How do I revert to a known stable kernel after an update prevents my Amazon EC2 instance from rebooting successfully?

2.    After you revert to a previous kernel, reboot the instance. Then, correct the issues on the corrupted kernel.

Dependency failed errors

Auto-mount failures caused by syntax errors in the /etc/fstab file can cause the instance to enter emergency mode. Also, if the Amazon Elastic Block Store (Amazon EBS) volume listed in the file is detached from the instance, then the instance boot process might enter emergency mode. If either of these problems occur, then the console output looks similar to the following:

-------------------------------------------------------------------------------------------------------------------
[[1;33mDEPEND[0m] Dependency failed for /mnt.
[[1;33mDEPEND[0m] Dependency failed for Local File Systems.
[[1;33mDEPEND[0m]
    Dependency failed for Migrate local... structure to the new structure.
[[1;33mDEPEND[0m] Dependency failed for Relabel all filesystems, if necessary.
[[1;33mDEPEND[0m] Dependency failed for Mark the need to relabel after reboot.
[[1;33mDEPEND[0m]
    Dependency failed for File System Check on /dev/xvdf.
-------------------------------------------------------------------------------------------------------------------

The preceding example log messages show that the /mnt mount point failed to mount during the boot sequence.

To prevent the boot sequence from entering emergency mode due to mount failures:

  • Add a nofail option in the /etc/fstab file for the secondary partitions ( /mnt, in the preceding example). When the nofail option is present, the boot sequence isn't interrupted, even if mounting of any volume or partition fails.
  • Add 0 as the last column of the /etc/fstab file for the respective mount point. Adding the 0 column disables the file system check, allowing the instance to successfully boot.

There are two methods you can use to correct the /etc/fstab file:

Important:

  • These procedures require a stop and start of your EC2 instance. Instance store data is lost when an instance is stopped and started. Be aware that if your instance is instance store-backed or has instance store volumes containing data, then the data is lost when the instance is stopped. For more information, see Determining the root device type of your instance.
  • If your instance is part of an Amazon EC2 Auto Scaling group, or if your instance is launched by services that use AWS Auto Scaling, such as Amazon EMR, AWS CloudFormation, AWS Elastic Beanstalk, and so on, then stopping the instance could terminate the instance. Instance termination in this scenario depends on the instance scale-in protection settings for your Auto Scaling group. If your instance is part of an Auto Scaling group, temporarily remove the instance from the Auto Scaling group before starting the resolution steps.
  • Stopping and starting the instance changes the public IP address of your instance. It's a best practice to use an Elastic IP address instead of a public IP address when routing external traffic to your instance.

Method 1: Run the AWSSupport-ExecuteEC2Rescue automation document

If your instance is configured for AWS Systems Manager, you can run the AWSSupport-ExecuteEC2Rescue automation document to correct boot issues. Manual intervention isn't needed when using this method. For information on using the automation document, see Walkthrough: Run the EC2Rescue tool on unreachable instances.

Method 2: Manually edit the file using a rescue instance

1.    Open the Amazon EC2 console.

2.    Choose Instances from the navigation pane, and then select the instance that's in emergency mode.

3.    Stop the instance.

4.    Detach the Amazon EBS root volume (/dev/xvda or /dev/sda1) from the stopped instance.

5.    Launch a new EC2 instance in same Availability Zone as the impaired instance. The new instance becomes your rescue instance.

6.    Attach the root volume you detached in step 4 to the rescue instance as a secondary device.

Note: You can use different device names when attaching secondary volumes.

7.    Connect to your rescue instance using SSH.

8.    Create a mount point directory for the new volume attached to the rescue instance in step 6. In the following example, the mount point directory is /mnt/rescue.

$ sudo mkdir /mnt/rescue

9.    Mount the volume at the directory you created in step 8.

$ sudo mount /dev/xvdf /mnt/rescue

Note: The device (/dev/xvdf, in the preceding example) might be attached to the rescue instance with a different device name. Use the lsblk command to view your available disk devices along with their mount points to determine the correct device names.

10.    After the volume is mounted, run the following command to open the /etc/fstab file.

$ sudo vi /mnt/rescue/etc/fstab

11.    Edit the entries in /etc/fstab as needed. The following example output shows three EBS volumes defined with UUIDs, the nofail option added for both secondary volumes, and a 0 as the last column for each entry.

------------------------------------------------------------------------------------------
$ cat /etc/fstab
UUID=e75a1891-3463-448b-8f59-5e3353af90ba  /  xfs  defaults,noatime  1  0
UUID=87b29e4c-a03c-49f3-9503-54f5d6364b58  /mnt/rescue  ext4  defaults,noatime,nofail  1  0
UUID=ce917c0c-9e37-4ae9-bb21-f6e5022d5381  /mnt  ext4  defaults,noatime,nofail  1  0  
------------------------------------------------------------------------------------------

12.    Save the file, and then run the umount command to unmount the volume.

$ sudo umount /mnt/rescue

13.    Detach the volume from the temporary instance.

14.    Attach the volume to original instance, and then start the instance to confirm that it boots successfully.


Did this article help you?

Anything we could improve?


Need more help?