Containers

Advice on mitigating the Apache log4j security issue for EKS, ECS, and Fargate customers

Note – AWS Fargate no longer supports the opt-in log4j Java hot patch capability for previously opted-in or new customers of Fargate as of 10-27-2023. Please ensure all containerised workloads leveraging the log4j package have been patched before scheduling them on to AWS Fargate.

CVE-2021-44228 (and subsequently CVE-2021-45046) describe a security issue found in the Apache Log4j 2 Java logging library versions 2.0-beta9 up to and including version 2.15.0. This issue uses the Java Naming and Directory Interface (JNDI), and allows a malicious actor to perform remote code execution on a vulnerable platform. The Amazon Web Services (AWS) security bulletin on this topic provides summary response information from various AWS services.

In this blog, we will discuss how Amazon Elastic Container Service (Amazon ECS) and Amazon Elastic Kubernetes Service (Amazon EKS) customers using Amazon EC2 and AWS Fargate to run their containerized applications can identify and mitigate CVE-2021-44228 and CVE-2021-45046 (the “log4j2 issue”).

Identify and Remediate

AWS strongly recommends that customers inspect their application container images and upgrade Log4j 2 to version 2.16 or higher to address this vulnerability as soon as possible. Amazon Elastic Container Registry (Amazon ECR) offers Enhanced Scanning with Amazon Inspector, which is designed to continually scan container images for known security issues, including container images containing CVE-2021-44228 and CVE-2021-45046. A 15-day trial for AWS accounts new to Inspector allows customers to use ECR enhanced scanning to detect for this log4j2 issue in their container images stored in ECR. This AWS security services blog post describes the log4j2 issue in more depth and discusses ways to detect and respond to it.

Mitigate

Mitigations use Amazon Linux packages containing the recently announced Hotpatch for Apache Log4j. Refer to this blog post to learn more about how this patch works. It’s important to note this patch is designed as a short-term, best effort mitigation until customers update Log4j 2 dependencies in their Java-based applications to at least Log4j version 2.16. This patch may help mitigate risk when updating dependencies is not immediately possible. Customers’ applications using container images that have already been updated to use Log4j 2 version 2.16 or higher should not apply the mitigations described below.

Amazon EKS customers using EC2

The Kubernetes team at AWS has open-sourced a project that packages the patch as a node agent which can be installed on any Kubernetes cluster as a DaemonSet. Running this DaemonSet will patch JVMs running in containers on Amazon Linux and Amazon Linux 2 as well as on the host. This tool may help you to mitigate risk when updating Log4j 2 application dependencies is not immediately possible. To use this node agent, visit the project on GitHub.

Amazon ECS customers using EC2

A package is available for Amazon Linux which, when applied, will continuously identify and apply a hot fix to Java processes including Java processes running in containers and nested containers. This new package can be installed on your Amazon ECS instances to help mitigate the log4j2 issue. There are three ways to apply this on ECS EC2 instances, which we describe below.

Installing on ECS EC2 instances using AWS Systems Manager

Customers can set up an AWS Systems Manager (SSM) Association to run the log4j hotpatch installation script against all their EC2 instances, which will include the ECS container instances as well. As SSM is a regional service, all steps below should be done in every region where ECS instances are running Java-based containers.

  • Onboard instances to SSM. Follow this documentation to do so.
  • At a high level:
    • Ensure the SSM Agent is installed on an instance. If you use Amazon Linux 2-based ECS-optimized AMI, you can skip this step as the SSM Agent comes included.
    • Make sure the AmazonSSMManagedInstanceCore IAM policy is included in the IAM role used for the EC2 instance profile.
    • These EC2 instances will appear in the AWS Systems Manager console under Instances & Nodes > Managed Instances.
    • Create an AWS CloudFormation stack with this template
AWSTemplateFormatVersion: "2010-09-09"
Resources:
  InstallLog4JHotPatchAgentAssociation:
    Type: AWS::SSM::Association
    Properties:
      AssociationName: "InstallLog4JHotPatch"
      Name:
        Ref: InstallLog4JHotPatchDocument
      MaxConcurrency: "1"
      Targets:
        - Key: "InstanceIds"
          Values:
            - "*"
  InstallLog4JHotPatchDocument:
    Type: AWS::SSM::Document
    Properties:
      Content:
        mainSteps:
          - "action": "aws:runShellScript"
            "name": "InstallLog4JHotPatch"
            "inputs":
              "runCommand":
              - "yum install -y log4j-cve-2021-44228-hotpatch"
        parameters: {}
        schemaVersion: "2.0"
      DocumentType: "Command"
    • This will create an SSM document that installs log4j hotpatch on EC2 instances, and create an SSM Association which will execute the SSM document against all EC2 instances, including new ones, launched from the AWS account. This ensures that all new instances added into your cluster through autoscaling will automatically have this patch applied.
    • Visit the AWS Systems Manager console under Instances & Nodes > State Manager to confirm the InstallLog4jHotPatch association has a Success status.

Installing on ECS EC2 instances using EC2 User Data

User Data is a short shell script that EC2 runs when launching an instance, see this documentation for more information. Add this command to your ECS EC2 instances’ User Data.

#!/bin/bash
yum install -y log4j-cve-2021-44228-hotpatch

Manual installation on ECS EC2 instances

As a break-glass option, customers can SSH into their ECS EC2 instances and manually install the patch.

$ sudo yum install -y log4j-cve-2021-44228-hotpatch

AWS Fargate customers

AWS Batch, Amazon ECS, and Amazon EKS customers using Fargate should contact AWS Support and request to opt-in to having the log4j Java hot patch applied to their accounts. This opt-in is per account and Region, so the request must include all account IDs and all Regions where the hot patch should be applied. Once an account is opted in for hot patching in a region, all new Fargate task/pod launches will be hot patched. Already running tasks/pods will need to be replaced.

As an aside, we chose to make this an opt-in mechanism to avoid potential impact to customer applications without consent, and to give customers a way to try this in non-production environments before opting in for accounts used for production.

AWS Batch customers

Given the ephemeral nature of EC2 and Fargate Batch jobs, updating container images to patch the log4j2 security issue is the recommended solution for AWS Batch customers. This is because short duration jobs, less than 30 minutes, will be missed by the hotpatch’s application logic when combined with the nature of continuous job dispatch from job queues and job placement on scaled EC2 instances.

For near-term mitigation when using Batch-launched EC2 instances, customers can apply the patch using Option 1 or 2 from the Amazon ECS customers using EC2 section above.

  • Option 1, patching via SSM, will work for existing and new EC2 instances launched by Batch
  • Option 2, using User Data, will work for new EC2 instances and can be applied via LaunchTemplate support in AWS Batch. An example is below:
==MYBOUNDARY==
Content-Type: text/cloud-config; charset="us-ascii"

packages:
- log4j-cve-2021-44228-hotpatch

==MYBOUNDARY==--

For AWS Batch customers with Fargate jobs and who would like to mitigate with the hotpatch, see AWS Fargate advice above.

AWS Elastic Beanstalk customers

AWS Elastic Beanstalk installs Log4j2 from the Amazon Linux default package repositories in its Tomcat platforms for Amazon Linux 1 and Amazon Linux 2. The versions of Log4j2 available in the Amazon Linux 1 and Amazon Linux 2 repositories are not affected by the log4j2 issue. If you have made configuration changes to your application’s use of log4j, or installed newer versions of log4j, then we recommend that you update your application’s code and upgrade to log4j2 version 2.16 or higher.

Elastic Beanstalk is releasing new platform versions that use the latest Amazon Linux default package repositories, which includes the log4j hotpatched JDK in new platform releases. For customers who have customized log4j installations as their application’s dependency, it is recommended to upgrade to the latest Beanstalk platform version to mitigate CVE-2021-44228 or CVE-2021-45046. Customers can use managed platform updates to automatically get this mitigation when it is available.

AWS App Runner customers

App Runner customers using Java with affected versions of log4j in their deployed services should rebuild their container images using updated log4j versions and redeploy their App Runner service.

Amazon ECS customers using EC2 or AWS Fargate and running Windows-based containers

Choose one of the following mitigation options that best suit your needs:

  • Update your existing Task Definitions for all containers that are using affected versions of Log4j2.
    • Via the Console, set: LOG4J_FORMAT_MSG_NO_LOOKUPS=true for each affected container within the Task.
    • Via the AWS Command Line or SDK update the “environment” section for each affected container within the Task as below:
"environment" : [
  {
    "name" : "LOG4J_FORMAT_MSG_NO_LOOKUPS", "value" : "true" }
  }
]
  • If updating the Task Definition is not possible you can update your existing Dockerfile in one of the following two ways.
    • Add the ENV command setting in your container before launching your application.
FROM openjdk:##-jdk
...
ENV LOG4J_FORMAT_MSG_NO_LOOKUPS=true
...
    • Or add the log4j2.formatMsgNoLookups JVM startup option when launching the application.
FROM openjdk:##-jdk
...
ENTRYPOINT ["java", "-Dlog4j2.formatMsgNoLookups=true", "...", "{EntryClass}"]

Bottlerocket customers

AWS-supported Bottlerocket images are not impacted by the log4j2 issue. As a minimal OS, it does not include Apache Log4j2 as part of the operating system. However, Bottlerocket customers may be using the logging library within their containers. To help mitigate impact on their Java-based containers, Bottlerocket 1.5.0 added the ability to hotpatch Java processes inside containers (Hotdog).

Hotpatching can be enabled for new instance launches by including the following settings in user data.

[settings.oci-hooks]
log4j-hotpatch-enabled = true

For existing hosts running the latest version of Bottlerocket, hotpatching can be enabled using the API client.

$ apiclient set oci-hooks.log4j-hotpatch-enabled=true

This feature is available for EKS, ECS and VMWare variants of Bottlerocket. Please upgrade your nodes to the latest Bottlerocket release (1.5.0) to use this feature. For more details, refer to our GitHub announcements.

With contributions from Vaishnavi Venkatesan, Sharanya Devaraj, Rashid Aga, and Mike Stefaniak.