How can I send user-data output to the console logs on an EC2 instance running RHEL 7 or RHEL 8?

Last updated: 2020-08-14

I'm trying to troubleshoot my Amazon Elastic Compute Cloud (Amazon EC2) Linux RHEL 7 or RHEL 8 instance bootstrap. How can I log the user-data invocation and then ship it to the console logs?

Short description

To troubleshoot issues on your EC2 instance bootstrap without having to access the instance through SSH, you can add code to your user-data bash script that redirects all the output both to the /var/log/user-data.log and to /dev/console. When the code is executed, you can see your user-data invocation logs in your console.

Note: This resolution is for RHEL 7 and RHEL 8 only. For information on Amazon Linux and Amazon Linux 2, see How can I send user-data output to the console logs on an EC2 instance running Amazon Linux or Amazon Linux 2?

Resolution

1.    Open the Amazon EC2 console.

2.    Launch a new instance and SSH into it.

3.    Edit the GRUB_CMDLINE_LINUX line in /etc/default/grub and change "console=ttyS0,115200n8 console=tty0" to "console=tty1 console=ttyS0".

RHEL 7:

# cat /etc/default/grub
GRUB_TIMEOUT=1
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="console=tty1 console=ttyS0 net.ifnames=0 rd.blacklist=nouveau crashkernel=auto"
GRUB_DISABLE_RECOVERY="true"

RHEL 8:

# cat /etc/default/grub
GRUB_TIMEOUT=1
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="console=tty1 console=ttyS0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295 crashkernel=auto"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

4.    Recreate the /boot/grub2/grub.cfg file:

RHEL 7:

# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.10.0-1062.1.2.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-1062.1.2.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-026767dbe06a4910a5ce3bd0def903c0
Found initrd image: /boot/initramfs-0-rescue-026767dbe06a4910a5ce3bd0def903c0.img
done

RHEL 8:

# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
done

5.    Stop the instance.

6.    Create a new AMI from the modified instance.

7.    Launch a new instance from the new AMI.

8.    Enter the following command to redirect the user-data output console:

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
  yum -y update
  echo "Hello from user-data!"

The following is the line that redirects the user-data output:

exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

The following is sample console output:

RHEL 7:

  subscription-manager.x86_64 0:1.24.26-3.el7_8
  subscription-manager-rhsm.x86_64 0:1.24.26-3.el7_8
  subscription-manager-rhsm-certificates.x86_64 0:1.24.26-3.el7_8
  sudo.x86_64 0:1.8.23-9.el7
  systemd.x86_64 0:219-73.el7_8.8
  systemd-libs.x86_64 0:219-73.el7_8.8
  systemd-sysv.x86_64 0:219-73.el7_8.8
  teamd.x86_64 0:1.29-1.el7
  tuned.noarch 0:2.11.0-8.el7
  tzdata.noarch 0:2020a-1.el7
  util-linux.x86_64 0:2.23.2-63.el7
  yum.noarch 0:3.4.3-167.el7
  yum-utils.noarch 0:1.1.31-54.el7_8

Replaced:
  iwl7265-firmware.noarch 0:22.0.7.0-72.el7

Complete!
+ echo 'Hello from user-data!'
Hello from user-data!

RHEL 8:

Installed:
  grub2-tools-efi-1:2.02-82.el8_2.1.x86_64

  kernel-4.18.0-193.13.2.el8_2.x86_64

  kernel-core-4.18.0-193.13.2.el8_2.x86_64

  kernel-modules-4.18.0-193.13.2.el8_2.x86_64

  linux-firmware-20191202-97.gite8a0f4c9.el8.noarch

Complete!
+
echo 'Hello from user-data!'
Hello from user-data!

Did this article help?


Do you need billing or technical support?