AWS Fargate now supports UDP load balancing with Network Load Balancer

From machine learning inference to gaming, from web hosting to batch processing, customers are using AWS Fargate to innovate faster and build products without maintaining servers. Many of you told us how important it is for you to run UDP-based applications in Fargate in AWS containers roadmap issue #445. User Datagram Protocol, more popularly known as UDP, is crucial for workloads like DNS, IoT, real-time media, and syslog that need low latency connection. Network Load Balancer (NLB), a fully managed Load Balancer that operates at the connection level (Layer-4) and is capable of handling millions of requests at ultra-low latencies, added support for UDP load balancing last year. Starting today, you can use NLB to distribute UDP traffic to applications running in Amazon ECS on Fargate. NLB support for Amazon EKS on Fargate is coming soon, you can track its progress in issue #617.

Design considerations for Network Load Balancer with UDP-based Fargate applications

Although the process of creating an NLB for UDP-based Fargate services is exactly the same as for TCP-based applications, below are some factors to consider:

Source IP address preservation

When you create a load-balanced ECS service that uses Fargate launch type, tasks register as NLB targets using their IP address. However, unlike using IP targets for TCP traffic, for UDP traffic NLB will preserve the source IP address. In other words, with Fargate, you don’t need to parse headers to get the source IP address. Another thing to note is that when handling UDP traffic, the load balancer selects a target Fargate task using a flow hash algorithm based on the protocol, source IP address, source port, destination IP address, and destination port. A UDP flow that has the same source and destination will consistently be routed to a single Fargate task during its lifetime.

Health checks

Given that health checks cannot be performed using UDP, you will need to use TCP-based health checks. If your application only uses UPD traffic, you can create a sidecar container that exposes a TCP port. For example, the sample application used in this post runs a sidecar NGINX container that listens on port 80.

Multi-protocol applications

If you have an application like DNS servers that listens on both TCP and UDP ports, you need to create two ECS services and register them with a multi-protocol target group (use TCP_UDP for both the NLB Listener and the target group.) You cannot expose both TCP and UDP ports in a single ECS service.

Network topology

For UDP-based load balancing, the NLB and Fargate tasks need to be in the same Amazon Virtual Private Cloud (VPC).

A sample UDP application using NLB and Fargate

This sample UDP-based application uses Fargate to run rsyslogd containers for syslog collection from Amazon EC2 instances. There’s an ECS service that runs two tasks. Each task runs two containers: one rsyslogd container that collects log and one NGINX container. The NGINX container responds to NLB health checks. In a real-world application, you should use a more reliable method for health checking. The rsyslogd containers write forwarded syslogs to an Amazon EFS volume. An NLB distributes traffic to the Fargate tasks. The EC2 instances are configured to forward syslogs to NLB’s DNS name.

You will need these things to follow along:

• AWS CLI
• A VPC with two subnets. You can use just one subnet if you’d like, you’ll need to edit the commands below. The subnets can be public or private.

Note the subnet ID of the subnets you plan to use. You will be referencing them throughout the tutorial.

Let’s start by setting a few environment variables. Change the variable to match your account:

export MYVPC=<<Your VPC ID>>
export MYSUBNET1=<<ID of a subnet in your VPC>>
export MYSUBNET2=<<ID of another subnet in the same VPC>>

Security groups for Fargate tasks

You’ll start by creating a security group that you will use for Fargate tasks.

aws ec2 create-security-group --description UDPLab-SyslogFargate \
    --group-name UDPLab-SyslogFargate --vpc-id $MYVPC

# Store the Fargate task Security Group ID in an environment variable
FargateSG=$(aws ec2 describe-security-groups  \
    --filters "Name=group-name,Values=UDPLab-SyslogFargate" "Name=vpc-id,Values=$MYVPC" \
    --query 'SecurityGroups[0].GroupId' \
    --output text)

Add a rule to allow syslog traffic from the CIDR the VPC and another to allow HTTP traffic, which allows the Network Load Balancer to perform health checks.

VPCCIDR=$(aws ec2 describe-vpcs --vpc-ids $MYVPC \
    --query "Vpcs[0].CidrBlock" \
    --output text)
    
# Allow syslog traffic from the VPC CIDR
aws ec2 authorize-security-group-ingress --group-id $FargateSG \
    --protocol udp \
    --port 514\
    --cidr $VPCCIDR

# Allow http traffic from the VPC CIDR
aws ec2 authorize-security-group-ingress --group-id $FargateSG \
    --protocol tcp \
    --port 80\
    --cidr $VPCCIDR

Create an Amazon EFS volume

Amazon Elastic File System (Amazon EFS) provides a simple, scalable, and fully managed elastic NFS file system. EFS provides massively-parallel shared access — multiple containers and EC2 instances can concurrently read and write a shared EFS volume. The rsyslogd processes running in Fargate tasks will store forwarded syslogs on an EFS volume.

To learn more about Fargate’s support for EFS, read Massimo Re’ Ferre’s post developers guide to using Amazon EFS with Amazon ECS and AWS Fargate.

Create a security group that will give Fargate tasks network access to the EFS volume. Allow NFS traffic from the security group you created in the previous step.

aws ec2 create-security-group --description EFS-for-Fargate \
    --group-name UDPLab-EFS-for-Fargate --vpc-id $MYVPC 

# Store the EFS Security Group ID in an environment variable
EFSSG=$(aws ec2 describe-security-groups  \
    --filters "Name=group-name,Values=UDPLab-EFS-for-Fargate" \
        "Name=vpc-id,Values=$MYVPC"  \
   --query 'SecurityGroups[0].GroupId' \
   --output text)

# Allow NFS traffic from the Fargate task Security Group
aws ec2 authorize-security-group-ingress --group-id $EFSSG \
    --protocol tcp \
    --port 2049 \
    --source-group $FargateSG

Create an EFS volume:

aws efs create-file-system \
    --creation-token UDPLab-EFS-For-Syslog \
    --performance-mode generalPurpose \
    --throughput-mode bursting \
    --tags Key=Name,Value=UDPLab-EFS-For-Syslog
    
# Store the filesystemid in an environment variable 
EFSFSID=$(aws efs describe-file-systems \
    --creation-token UDPLab-EFS-For-Syslog \
    --query "FileSystems[*].FileSystemId" \
    --output text)

Create mount targets for the volume in both the subnets:

# Subnet 1
aws efs create-mount-target \
--file-system-id $EFSFSID \
--subnet-id  $MYSUBNET1 \
--security-group $EFSSG

# Subnet 2
aws efs create-mount-target \
--file-system-id $EFSFSID \
--subnet-id  $MYSUBNET2 \
--security-group $EFSSG

If you want to add more than two Availability Zones, repeat this step for each Availability Zone.

Create a Network Load Balancer

Create an internal Network Load Balancer specifying the two subnets:

aws elbv2 create-load-balancer --name UDPLab-SyslogNLB \
    --type network \
    --scheme internal \
    --subnets $MYSUBNET1 $MYSUBNET2
    
# Store the NLB ARN in an environment variable
SyslogNLB=$(aws elbv2 describe-load-balancers \
    --name UDPLab-SyslogNLB \
    --query "LoadBalancers[*].LoadBalancerArn" \
    --output text)

Create a target group for your NLB. The target type will be IP, the protocol will be UDP, and the port will be 514 (syslog port). TCP port 80 will be used for health checks. Since UDP is connectionless, it cannot be used for checking the health of the Fargate task. The NGINX container in the Fargate task will listen on port 80 and respond to NLB’s health checks:

aws elbv2 create-target-group --name UDPLab-SyslogTG\
    --protocol UDP \
    --port 514 \
    --target-type ip \
    --health-check-protocol TCP \
    --health-check-port 80 \
    --vpc-id $MYVPC

# Store the Target Group in an environment variable     
SyslogTG=$(aws elbv2 describe-target-groups \
    --name UDPLab-SyslogTG \
    --query "TargetGroups[*].TargetGroupArn" \
    --output text)

Create a listener for the NLB, which is a process that checks for connection requests, using the protocol and port that you configure. The rules that you define for a listener determine how the load balancer routes requests to the targets in one or more target groups. You will create a listener that listens on port 514/UDP.

aws elbv2 create-listener \
    --load-balancer-arn $SyslogNLB \
    --protocol UDP \
    --port 514  \
    --default-actions \
    Type=forward,TargetGroupArn=$SyslogTG

Create an IAM role for Fargate tasks

If you have never used ECS, you will also need to create an ECS task execution role:

cat > /tmp/task-execution-assume-role.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ecs-tasks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

aws iam create-role --role-name UDPLab-ecsTaskExecutionRole \
    --assume-role-policy-document file:///tmp/task-execution-assume-role.json
    
# Attach the Amazon Managed Policy    
aws iam attach-role-policy --role-name UDPLab-ecsTaskExecutionRole \
    --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
    
# Store the ECS task Execution Role in an environment variable
ECSTaskExecRole=$(aws iam get-role --role-name UDPLab-ecsTaskExecutionRole \
    --query "Role.Arn" \
    --output text)

Fargate tasks will need permission to write to the EFS volume, create an IAM role for tasks, and attach AmazonElasticFileSystemClientReadWriteAccess policy to the role. This tutorial doesn’t use this policy because the EFS volume you created doesn’t limit filesystem access using IAM. Read using IAM to control NFS access to Amazon EFS to learn more about using IAM to restrict access to EFS.

cat > /tmp/task-role.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ecs-tasks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

aws iam create-role --role-name UDPLab-TaskRole \
    --assume-role-policy-document file:///tmp/task-role.json
    
    
aws iam attach-role-policy --role-name UDPLab-TaskRole \
    --policy-arn arn:aws:iam::aws:policy/AmazonElasticFileSystemClientReadWriteAccess

# Store the ECS task Role in an environment variable
ECSTaskRole=$(aws iam get-role --role-name UDPLab-TaskRole --query "Role.Arn" --output text)

Create a task definition

You will need to create an ECS task definition. A task definition is required to run Docker containers in Amazon ECS.

This task definition specifies:

• Two containers:
  • syslog collector → Port 514/udp exposed
  • NGINX → Port 80/tcp exposed (for health checks only)
• The EFS volume created above
• CPU: 512
• Memory: 1GB

Create a file that holds the task definition:

cat > /tmp/task-definition.json << EOF
{
  "executionRoleArn": "$ECSTaskExecRole",
  "containerDefinitions": [
    {
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/Syslog",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs"
        }
      },
      "portMappings": [
        {
          "hostPort": 514,
          "protocol": "udp",
          "containerPort": 514
        }
      ],
      "mountPoints": [
        {
          "containerPath": "/var/log/remote",
          "sourceVolume": "Syslog"
        }
      ],
      "image": "realz/fargate-syslog:latest",
      "name": "Syslog"
    },
    {
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/Syslog",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs"
        }
      },
      "portMappings": [
        {
          "hostPort": 80,
          "protocol": "tcp",
          "containerPort": 80
        }
      ],
      "image": "nginx",
      "name": "NGINX"
    }
  ],
  "memory": "1024",
  "cpu": "512",
  "taskRoleArn": "$ECSTaskRole",
  "family": "Syslog-EFS",
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "networkMode": "awsvpc",
  "volumes": [
    {
      "efsVolumeConfiguration": {
        "fileSystemId": "$EFSFSID",
        "authorizationConfig": {
          "iam": "DISABLED"
        },
        "transitEncryption": "DISABLED",
        "rootDirectory": ""
      },
      "name": "Syslog"
    }
  ]
}
EOF

Register the task definition:

aws ecs register-task-definition --cli-input-json file:///tmp/task-definition.json

Create ECS cluster and service

An ECS cluster is a logical grouping of tasks or services. Create an ECS cluster.

aws ecs create-cluster --cluster-name Syslog 

Create an ECS service. An ECS service enables you to run and maintain a specified number of instances of a task definition simultaneously in an ECS cluster. If any of your tasks should fail or stop for any reason, the ECS service scheduler launches another instance of your task definition to replace it in order to maintain the desired number of tasks in the service. You will be running this ECS service behind the NLB you created earlier.

aws ecs create-service --cluster Syslog \
    --service-name syslog \
    --task-definition Syslog-EFS:1 \
    --desired-count 2 \
    --launch-type FARGATE \
    --platform-version 1.4.0 \
    --network-configuration "awsvpcConfiguration={
        subnets=[$MYSUBNET1,
            $MYSUBNET2],
        securityGroups=[$FargateSG],
        assignPublicIp=ENABLED}" \
    --load-balancers "targetGroupArn=$SyslogTG,
        containerName=Syslog,
        containerPort=514"

You need to assign public IP address to the tasks because the container image is hosted on Docker Hub. You can host the container image on ECR if you’d like to avoid assigning a public IP address.

It may take a few seconds before the tasks start. When you see two tasks running, you can proceed to the next step. To get the number of running tasks, run:

aws ecs describe-services --services syslog \
    --cluster Syslog \
    --query "services[0].deployments[0].runningCount"

Configure EC2 instances to forward syslog

Now you need a few EC2 instances to generate syslogs. Create an EC2 instance and configure the syslog to forward logs to the NLB DNS name. Copy the address of the syslog NLB:

aws elbv2 describe-load-balancers \
    --name UDPLab-SyslogNLB \
    --query "LoadBalancers[*].DNSName" \
    --output text

Edit /etc/rsyslog.conf on EC2 instance and create a forwarding rule with the NLB DNS name.

Notice the highlighted line: if you prefix the destination with an “@”, the traffic is sent as UDP, whereas prefixing the destination with “@@” will use TCP.

If you are using a Linux distribution with systemd, configure journald to forward logs to syslog. Edit /etc/systemd/journald.conf and comment ForwardToSyslog=yes.

Restart rsyslog and systemd-journald services. On Linux distributions with systemd:

sudo systemctl restart rsyslog systemd-journald

In distributions without systemd:

sudo /etc/init.d/rsyslog restart

View logs

The Fargate containers write logs to the shared EFS volume. To view the logs, you can mount the EFS Volume on an EC2 instance.

?Before you can mount the EFS volume, you have to add the security group of the EC2 instance to the EFS mount target(s):

# Determine the Security Group attached to your EC2 Instance
EC2SGName=$(curl http://169.254.169.254/latest/meta-data/security-groups)

# Get the security group id of any of the Security Groups from the 
# previous command
EC2SG=$(aws ec2 describe-security-groups \
    --filters "Name=group-name,Values=$EC2SGName" \
        "Name=vpc-id,Values=$MYVPC" \
    --query 'SecurityGroups[0].GroupId' \
    --output text)

aws ec2 authorize-security-group-ingress --group-id $EFSSG \
    --protocol tcp \
    --port 2049 \
    --source-group $EC2SG

Mount the EFS volume:

# Install * *amazon-efs-utils package
sudo yum install -y amazon-efs-utils

sudo mkdir /efs-syslog
sudo mount -t efs $EFSFSID:/ /efs-syslog

The /efs-syslog directory should have a directory for each Fargate task in the syslog service and any EC2 instances forwarding syslog.

Scaling the syslog service

In a real-world scenario, the syslog service should automatically scale based on resource consumption. To learn more scaling services in ECS, read service auto scaling.

Cleanup

Run these commands to terminate the resources you created in this tutorial:

# Delete the ECS service
aws ecs delete-service --service syslog --cluster Syslog --force
# Delete the NLB
aws elbv2 delete-load-balancer --load-balancer-arn $SyslogNLB
# Delete the target group
aws elbv2 delete-target-group --target-group-arn $SyslogTG
# Delete EFS mount Target #1
aws efs delete-mount-target --mount-target-id $(aws efs describe-mount-targets --file-system-id $EFSFSID --query "MountTargets[0].MountTargetId" --output text) 
# Delete EFS mount Target #2
aws efs delete-mount-target --mount-target-id $(aws efs describe-mount-targets --file-system-id $EFSFSID --query "MountTargets[0].MountTargetId" --output text) 
# Wait a few seconds before deleting the EFS volume 
aws efs delete-file-system  --file-system-id $EFSFSID
# Delete the EFS security group
aws ec2 revoke-security-group-ingress --group-id $EFSSG --ip-permission "`aws ec2 describe-security-groups --output json --group-ids $EFSSG --query "SecurityGroups[0].IpPermissions"`"
aws ec2 delete-security-group --group-id $EFSSG 
# Delete the EFS Fargate group
aws ec2 revoke-security-group-ingress --group-id $FargateSG --ip-permission "`aws ec2 describe-security-groups --output json --group-ids $FargateSG --query "SecurityGroups[0].IpPermissions"`"
aws ec2 delete-security-group --group-id $FargateSG 
# Deregister the task definition 
aws ecs deregister-task-definition --task-definition Syslog-EFS:1
# Delete the ECS cluster
aws ecs delete-cluster --cluster Syslog 
# Delete IAM roles
aws iam detach-role-policy --role-name UDPLab-TaskRole --policy-arn arn:aws:iam::aws:policy/AmazonElasticFileSystemClientReadWriteAccess
aws iam delete-role --role-name UDPLab-TaskRole
aws iam detach-role-policy --role-name UDPLab-ecsTaskExecutionRole --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
aws iam delete-role --role-name UDPLab-ecsTaskExecutionRole

Conclusion

Fargate support for UDP NLB is available starting today in the US East (N. Virginia), US West (Oregon), Europe (Ireland), and Asia Pacific (Tokyo) regions. Find out more about service load balancing in the ECS documentation.

Before parting, let us tell you a UDP joke —

Why did the UDP packet refuse to wash its hands for 20 mississippis?
…
It doesn’t do handshakes!