Dean shows you how to update
GRUB to load a previous kernel on
an EC2 instance that fails to reboot

Dean_CPT1018

How do I revert to a stable kernel after an update prevents my Amazon Elastic Compute Cloud (Amazon EC2) instance from rebooting successfully?

If you perform a kernel update to your Linux EC2 instance but the kernel is now corrupt, the EC2 instance won't reboot. Also, you can't use SSH to connect to the impaired instance. However, you can create a temporary rescue instance, remount your EBS volume on the rescue instance, and then alter the kernel parameters to the GRUB to remove the corrupt kernel.

Important: Do not perform this procedure if your instance is an instance store-backed instance—because this recovery procedure requires a stop and start of your instance, any data will be lost. For more information, see Determining the Root Device Type of Your Instance.

Attach the root volume to a rescue EC2 instance

1.    Create an EBS snapshot of the root volume. For more information, see Creating an Amazon EBS Snapshot.

2.    Open the Amazon EC2 console.

Note: Be sure that you are in the correct AWS Region.

3.    Choose Instances from the navigation pane, and then choose the impaired instance.

4.    Choose Actions, choose Instance State, and then choose Stop.

5.    In the Description tab, under the Root device, choose /dev/sda1, and then choose the EBS ID.

Note: The Root device differs by the AMI, but the /dev/xvda or /dev/sda1 are always reserved for the root device. For example, Amazon Linux 1 and 2 indicate /dev/xvda, while other distributions, such as Ubuntu 14, 16, 18, CentOS 7, and RHEL 7.5, are set as /dev/sda1.

6.    Choose Actions, choose Detach Volume, and then choose Yes, Detach. Note the Availability Zone.

7.    Launch a rescue EC2 instance in the same Availability Zone.

Note: Depending upon the product code, you might be required to launch an EC2 instance of the same OS type. For example, if the impaired EC2 instance is a paid RHEL AMI, you must launch an AMI with the same product code. For more information, see Getting the Product Code for Your Instance.

8.    After the rescue instance has launched, choose Volumes from the navigation pane, and then choose the detached root volume of the impaired instance.

9.    Choose Actions, and then choose Attach Volume.

10.     Choose the rescue instance ID (id-xxxxx), and then set an unused device. For our example, we're using /dev/xvdb.

Mount the volume of the impaired instance

1.    Use SSH to connect to the rescue instance.

2.    Run the lsblk command to view your available disk devices.

lsblk

The following is an example of the output.

NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
xvda    202:0     0   15G  0 disk
└─xvda1 202:1     0   15G  0 part /
xvdb    202:0     0   15G  0 disk
    └─xvdb1 202:1 0   15G  0 part

3.    Create a mount directory, and then mount the /dev/xvdb1 partition there. For more information, see Making an Amazon EBS Volume Available for Use on Linux.

sudo mkdir /mount
sudo mount /dev/xvdb1 /mount

You can now access the data of the impaired instance through the mount directory.

4.    Mount /dev, /dev/shm, /proc, and /sys of the rescue instance to the same paths as the newly mounted volume.

sudo mount -o bind /dev /mount/dev
sudo mount -o bind /dev/shm /mount/dev/shm
sudo mount -o bind /proc /mount/proc 
sudo mount -o bind /sys /mount/sys

5.    Call the chroot function to change into the mount directory.

sudo chroot /mount

Update the default kernel in the GRUB bootloader

The current, corrupt kernel is in position 0 (zero) in the list. The last stable kernel is in position 1. To replace the corrupt kernel with the stable kernel choose one of the following procedures, based on your distro:

  • GRUB1 (Legacy GRUB) for Red Hat 6 and Amazon Linux
  • GRUB2 for Ubuntu 14 LTS and 16.04
  • GRUB2 for RHEL 7.5 and Amazon Linux 2

GRUB1 (Legacy GRUB) for Red Hat 6 and Amazon Linux 1

Use the sed command to replace the corrupt kernel with the stable kernel in the /boot/grub/grub.conf file.

sudo sed -i '/^default/ s/0/1/' /boot/grub/grub.conf

GRUB2 for Ubuntu 14 LTS and 16.04

1.    Replace the corrupt GRUB_DEFAULT=0 default menu entry with the stable GRUB_DEFAULT=saved value in the /etc/default/grub file.

sed -i 's/GRUB_DEFAULT=0/GRUB_DEFAULT=saved/g' /etc/default/grub

2.    Update grub to recognize the change.

sudo update-grub

3.    Run the grub-set-default command so that the stable kernel loads at the next reboot. In this example, grub-set-default is set to 1 in position 0.

sudo grub-set-default 1

GRUB2 for RHEL 7.5 and Amazon Linux 2

1.    Replace the corrupt GRUB_DEFAULT=0 default menu entry with the stable GRUB_DEFAULT-saved value in the /etc/default/grub file.

sed -i 's/GRUB_DEFAULT=0/GRUB_DEFAULT=saved/g' /etc/default/grub

2.    Update grub to regenerate the /boot/grub2/grub.cfg file.

sudo grub2-mkconfig -o /boot/grub2/grub.cfg

3.    Run the grub2-set-default command so that the stable kernel loads at the next reboot. In this example grub2-set-default is set to 1 in position 0.

sudo grub2-set-default 1

4.    Type exit to leave the chroot environment.

Detach the root volume from the rescue instance and attach it to the impaired instance

1.    From the Amazon EC2 console, choose Instances, and then choose the rescue instance.

2.    Choose Actions, choose Instance State, choose Stop, and then choose Yes, Stop.

3.    Detach the root volume vol-xxx from the rescue instance.

4.    Attach the root volume to the impaired instance as the root volume (/dev/sda1), and then start the instance.

Note: Depending on the distribution, the root volume will vary. For Amazon Linux 1 and 2 the root volume must be /dev/xvda. If it's not, you'll receive an error. Likewise, if the distro is RHEL, CentOS, or Ubuntu, then the root volume must be /dev/sda1.

The stable kernel will now load and your instance will reboot.


Did this page help you? Yes | No

Back to the AWS Support Knowledge Center

Need help? Visit the AWS Support Center

Published: 2019-01-15