Building an Automotive Embedded Linux Image for Edge and Cloud Using Arm-based Graviton Instances, Yocto Project, and SOAFEE

Embedded software developers in the automotive industry have used a traditional embedded development methodology that has been largely unchanged for decades. The embedded target system that the software is being written for is by nature resource constrained with limited memory, compute, and input/output capacity. Additionally, embedded development systems are often expensive and a scarce commodity for development teams working on brand new electronic control units (ECUs) and system-on-chip (SoC) architectures. This scarcity has become further apparent with the recent global chip shortages in the automotive industry.

This means software developers avoid developing software directly on the embedded platform itself and instead rely on developing software on a powerful development machine or host. Compilers and toolchains used to create executable software for development and testing on the host machine cannot be used directly for deployment on the embedded target. Many times, the target-embedded operating system (OS) itself cannot be used on the host machine. Developers rely on cumbersome OS emulation tools on their host machine and the process of cross-compiling, which uses a special compiler to create executable code for the target system. Once that code is on the development system, a final integration and validation testing can be performed, but scaling is limited to the number of physical hardware platforms. Typical development, integration, and validation workflows for embedded systems look like the following today:

This blog will introduce a building block toward a novel automotive-software development concept that helps developers create, test, and debug natively compiled software using the cloud, greatly simplifying this workflow. This concept is often referred to as environmental parity, which you can learn more about in this whitepaper from Arm and Amazon Web Services (AWS), as well as in Kevin Hoffman’s book, Beyond the Twelve-Factor App.

“The purpose of…environment parity is to give your team and your entire organization the confidence that the application will work everywhere.” —Kevin Hoffman

Environmental parity is a major element in achieving the objective of cloud-native-software-defined vehicles.

Arm, AWS, and SOAFEE

Arm technology is defining the future of computing. Arm’s energy-efficient processor designs and software platforms have enabled advanced computing in more than 225 billion chips and our technologies securely power products from the sensor to the smartphone and the supercomputer. Together with 1,000+ technology partners, we are enabling artificial intelligence to work everywhere, and in cybersecurity, we are delivering the foundation for trust in the digital world – from chip to cloud. Arm architectures are the most diffused inside modern vehicles, from infotainment units to body control, and much more, including very delicate functional safety related compute workloads.

In 2021 Arm, AWS, and other founding members announced the Scalable Open Architecture for Embedded Edge (SOAFEE) Special Interest Group, which brings together automakers, semiconductor, and cloud technology leaders to define a new open-standards based architecture to implement the lowest levels of a software-defined vehicle stack. It delivers a reference implementation that enables cloud-native technologies like microservices, container, and orchestration systems to be combined with automotive functional safety for the first time, preserving environmental parity.

ISA Parity from Cloud to Embedded Edge with SOAFEE

AWS and Arm have a long history. This includes the use of 64-bit processor inside AWS custom-built SoCs that are used in AWS Graviton processors, which deliver optimal price performance for cloud workloads. By having these processor options on an increasing number of instance types, automotive developers can develop and run cloud-native applications and toolchains using the same Arm intellectual property (IP) and tools that are also used in embedded automotive platforms such as the AVA Developer Platform from ADLINK, an AWS Partner.

By using this IP in the cloud and the edge, developers can achieve the first level of environmental parity at the instruction set architecture (ISA) level. However, to achieve even higher levels of parity, we will need an OS, abstraction layer, and orchestration layers that are automotive grade, which SOAFEE is working to support. The below diagram shows different levels of parity and developer personas associated with those approaches.

Achieving such levels of parity will facilitate completely new workflows for the development and validation of embedded software, as we will explain in the next sections.

Using ISA parity to facilitate OS-level parity

As a reference OS to demonstrate the concepts explained in this blog post, we will create a custom Linux distribution using the Yocto Project, an open-source collaboration project that helps developers create custom Linux-based systems regardless of the hardware. It is used by many embedded projects, including the Automotive Grade Linux initiative that is part of the Linux Foundation as well as the SOAFEE reference implementation.

To enable OS-level parity, Arm and AWS collaborated to create an Amazon Machine Image (AMI) for Graviton instances, based on the Yocto Project. An AMI is a template that contains a software configuration, including OS and application software. That can be used to launch an Amazon EC2 instance, which is a copy of the AMI running as a virtual server in the cloud. The developed AMI also includes Arm’s Edge Workload Abstraction and Orchestration Layer (EWAOL), an open-source reference implementation of the SOAFEE architecture, which facilitates further application, container, and OS environmental parity.

EWAOL provides a standards-based framework that uses containers for the deployment and orchestration of embedded applications. This approach helps developers to start writing and testing code in the cloud, significantly improving and scaling the workflow. More specifically, SOAFEE facilitates

• the cloud implementation of the entire embedded software stack, from the embedded OS up, and not just of the unit of software in development; and
• seamless portability from cloud to embedded edge—no more cross compilation or emulation (and related issues, like compilation errors or performance degradation).

Visit the Arm GitLab for more information on EWAOL.

Now that we’ve introduced the concept of OS-level and ISA-level parity, we can now modify the embedded developers’ workflow to cut many of the steps that are now no longer required as shown in the below diagram:

For the remainder of this blog post, we will present a tutorial detailing how to create your own custom Linux-based AMI for AWS Graviton–based instances using the Yocto Project. The tutorial is not intended just to teach how to deploy an already-created AMI on AWS but to provide developers with a methodology to bring their own Linux images used for embedded systems to the cloud.

How to create a custom Linux AMI for AWS Graviton processors using the Yocto Project

This guide will describe how to create a custom Linux AMI that can be used to launch an Amazon EC2 instance. It is expected that you will already have access to an AWS account with appropriate permissions to create the resources required. If not, create/activate an account.

Before getting started, please review the detailed instructions available on GitLab. This repository contains the latest information and instructions on how to create a custom Linux AMI including EWAOL.

There are five main steps to creating a custom AMI with Yocto. We will go through them step by step and build out the architecture that is shown below:

1. Configure AWS resources

First, we’ll need to do some AWS account setup and configuration using AWS CloudFormation, which lets you model, provision, and manage AWS and third-party resources by treating infrastructure as code. AWS CloudFormation is used here to provision AWS resources using a configuration file to save you time.

1. Download this AWS CloudFormation from the GitLab repo to your local machine.
2. Go to the AWS CloudFormation dashboard.
3. Click on “Create Stack”
4. Select “Upload a template file,” Choose the template file you downloaded in step 1 and click “Next” to proceed.
5. Enter “yocto-build” for the stack name. Switch create S3 bucket to “yes” and enter any unique bucket name. Select “Next” to proceed.
6. Keep this page as defaults and select “Next.”
7. Finally, review the settings, acknowledge the Identity and Access Management (resource creation, and select “Create stack.”
8. This should take a few minutes to create. When you see “CREATE_COMPLETE” in the stack status, you can move on to the next step.

2. Create a Yocto build Amazon EC2 instance

We’ll now create and Secure Shell (SSH) into an Amazon EC2 instance that will be used as our development environment for the compilation and building of our custom Linux image using the Yocto Project.

1. Go to the Amazon EC2 dashboard.
2. Click “Instances” in the sidebar menu.
3. Click the orange “Launch Instances” button.
4. Select “Ubuntu Server 20.04 LTS” with “64-bit (Arm)”
5. Select the “c6g.4xlarge” instance type. The c6g.4xlarge instance type is chosen because it is a CPU-optimized instance that will be used for compilation, but other instance types could be used based on your requirements.
6. Create and download a new keypair. You will use this key to encrypt your SSH session later.
7. Under “Create security group,” select “Allow SSH Traffic.” In the drop-down menu, select “My IP.”
8. Under “Configure Storage,” adjust the root volume to have 150GiB of gp2 (General Purpose SSD).
9. Under “Advanced details,” we’ll need to select our “IAM instance profile” that was created in the cloudformation step to provide our instance with the necessary AWS permissions for the following steps. It should start with “yocto-build-VMBuilderEC2Role-” followed by random characters.
10. Leave all of the remaining settings to their defaults and select “Launch Instance”.
11. Finally, you can select the “View Instances” button and wait for your instance to get to the “running state.”

3. Build your custom Linux image using the Yocto Project

Next, you should SSH into the build instance that was created in step 2. You can use any SSH client you would like to connect to this instance.

1. Find the SSH command by selecting the instance in the list and clicking on the “Connect” button.
   1. Using your SSH client, connect to the server using the commands shown. For example: ssh -I “keypair.pem” ubuntu@ec2-[ip-address].compute-1.amazonaws.com.
2. Install build dependencies as noted in the GitLab Repository.
3. Complete the “Building EWAOL” section from the same repository. This repository configures a basic Yocto Project Linux distribution that includes cloud-init, meta-ewaol, meta-aws, and SSH used to support our AMI.

Note: Spend time examining the YAML config file and make any modifications needed specific to your project.

4. After completing the “kas build” step above, it should take approximately 50 minutes to complete the build of our new Linux distribution.
5. While this completes, we optionally now dive deeper at the YAML file and kernel configuration used in this project.

(Optional Deep Dive) Explanation of kas YAML and kernel configuration

This YAML file describes to kas how Yocto Project should construct our Linux image.

Header

header:

  version: 10

  includes:

    - repo: meta-ewaol

      file: meta-ewaol-config/kas/baremetal.yml

This section tells kas which version of the file schema we are using, along with some other YAML files to include from the EWAOL repo (explained later). These included files load in the base configuration of EWAOL and some standard Arm machine definitions.

Repositories

repos:
  meta-arm:
    url: https://git.yoctoproject.org/git/meta-arm
    path: layers/meta-arm
    refspec: kirkstone
    layers:
      meta-arm: included
      meta-arm-toolchain: included
  meta-ewaol:
    url: https://gitlab.com/Linaro/ewaol/meta-ewaol
    path: layers/meta-ewaol
    refspec: v1.0
  meta-aws:
    url: https://github.com/aws4embeddedlinux/meta-aws.git
    path: layers/meta-aws
    refspec: kirkstone
  meta-virtualization:
    refspec: kirkstone
  meta-ewaol-ext:
    path: meta-ewaol-ext
    refspec: kirkstone
  meta-openembedded:
    refspec: kirkstone
  poky:
    refspec: kirkstone

This section tells kas which Yocto layers we want to use, where to download them, which versions to use, and where to store them. In this example, we are using meta-ewaol, the reference SOAFEE implementation layer, and aws-meta, which provides recipes for AWS edge software capabilities. In the header, “ewaol-base.yml” was included, and it contains many more repositories. This file also loads in ‘poky’ and ‘meta-openembedded,’ which are the standard Yocto layers. In addition, the meta-ewaol-ext layer adds customizations to the cloud-init and the OpenSSH recipes by using patches and bbappend files.

Machine

machine: generic-arm64

This line tells Yocto which machine we want to target our image for. In this example ‘generic-arm64’ is used. It creates an image that can boot on most Arm devices that support a generic boot mechanism (UEFI in this case. You will see this set for the AMI image later.).

Custom config

local_conf_header:

meta-custom: |
FILESEXTRAPATHS_prepend_pn-linux-yocto = “${TOPDIR}/../kernelconfig/:”
SRC_URI_append_pn-linux-yocto = ” file://gravitonKernelConfigs.cfg “

INHERIT += “extrausers”
# Hardening: Locking the root password. Creating the ewaol without password for ssh key-based login only
EXTRA_USERS_PARAMS = “usermod -L root; useradd -p ‘*’ ewaol”

EXTRA_IMAGE_FEATURES_append = “ssh-server-openssh”
# Forcing removal of debug-tweakes as ewaol includes it in all targets by default and that leads to reversing some sshd_config hardening done in our bbappend when do_rootfs runs
EXTRA_IMAGE_FEATURES_remove = “debug-tweaks”
IMAGE_INSTALL_append = ” rng-tools awscli cloud-init cloud-init-systemd e2fsprogs e2fsprogs-resize2fs e2fsprogs-tune2fs e2fsprogs-e2fsck e2fsprogs-mke2fs parted sudo sudo-sudo openssh-sftp-server”
IMAGE_FSTYPES += ” wic wic.vhdx”

This section configures custom config settings specific for this example:

• Adds cloud-init, which will be used to configure the AMI to the specific Amazon EC2 instance at boot time
• Adds in some extra kernel config
• Adds some extra features like an SSH server and the AWS Command Line Interface (AWS CLI), a unified tool to manage AWS services

Target

Target:

– ewaol-image-docker

This tells Yocto which target images to build. In this case, we only want it to build the docker image (not the podman image).

Kernel configuration fragment

To configure Yocto to work on AWS Graviton2 processors, a few additional kernel configurations must be activated. They are included in gravitonKernelConfigs.cfg.

• • CONFIG_BLK_DEV_NVME is required for the NVMe storage to work.
  • CONFIG_ENA_ETHERNET is required for working networking.
  • CONFIG_SERIAL_8250_PCI is required for the serial port.

4. Creating the AMI from image file

1. Now that the image is built, we need to create the AMI itself.
2. Follow the instructions from the GitLab repository to run the create-ami.sh script.
3. This will automatically generate the new AMI in your AWS Account.
4. Finally, you no longer need this build Amazon EC2 instance, so you can shut it down:
   “`
   $ sudo poweroff
   “`

5. Launch EC2 instance using the new AMI

Launching an instance using your newly created AMI is like setting up the EC2 instance earlier in this guide.

1. Go to the Amazon EC2 dashboard.
2. Open the AMI pane by clicking AMI in the left-hand menu.
3. Select the AMI created earlier.
4. Click the “launch instance from image” button. The following screens should be familiar from the first part of this guide.
5. Select the same keypair in the drop-down menu created previously.
6. Under “Create security group,” select “Allow SSH Traffic” and in the drop down select “My IP.”
7. You can leave the default settings for the other instance settings and click “Launch instance.”
8. Now we will connect to our custom Linux instance using our SSH utility. Find the SSH command by selecting the instance in the list and clicking on the “Connect” button. However, you must modify the username yourself to be “ewaol” as shown below.
   • For example: ssh -I “keypair.pem” ewaol@ec2-[ip-address].compute-1.amazonaws.com.
9. Let’s confirm that the instance is the one that we built using the Yocto Project with the following command:
   • “`
     $ uname –r
     “`
     • The output of the command should display the following, where x is the version number:
       “`
       x.x.x-yocto-standard
       “`
10. Finally, be sure to delete the Stack and all resources created in this tutorial when you no longer need them to avoid incurring future costs.

Congrats! You now have been able to build and launch your own custom Linux image using the Yocto Project that now can be used in the cloud for development as well as verification and validation.

If you encounter any issues with your custom images, you can access the instances serial console to debug. This is not activated by default. Details can be found here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-serial-console.html

The future of automotive is software defined

In this blog, we have covered the high-level concept of environmental parity and what Arm, AWS, and other partners are working toward: building out a software-defined system and accelerating automotive software development. We also provided a tutorial on how you can create a custom AMI using the Yocto Project for your own development workflow using Arm-based AWS Graviton processors.

If you would like to go deeper into this concept, visit this hands-on workshop detailing how to create an automated software development pipeline for automotive development using the AMI created in this blog and demonstrating environmental parity.

For more information about the Scalable Open Architecture for Embedded Edge (SOAFEE) project, visit soafee.io.