AWS Compute Blog
Architecting for seamless on-premises connectivity with AWS Outposts servers
This post is written by Mark Nguyen, Principal Solutions Architect, AWS and Ryan Fillis, Solutions Architect, AWS.
AWS Outposts brings native AWS services, infrastructure, and operating models to virtually any data center, co-location space, or on-premises facility. Deploying Outposts servers in your environment necessitates additional considerations regarding local network connectivity and Amazon Elastic Compute Cloud (Amazon EC2) instance networking. This post demonstrates the scalability of Outposts servers through automation and the deployment of Amazon EC2 network interfaces. This reduces the number of manual steps required to configure an Outposts server.
This post details physically connecting your servers to your Local Area Network (LAN) and the networking options available for EC2 instances running on Outposts. We cover the physical cabling options, virtual networking components such as VPCs and subnets, and walkthrough an example setup for an EC2 instance with a user-data script to route traffic locally over your on-premises network.
This post assumes that you have some familiarity with Outposts servers. If you would like a general refresher, observe What is AWS Outposts. For more information about how to provision your Outposts server, see Installing an AWS Outposts server.
Basic Amazon EC2 networking using a single interface
When launching an EC2 instance on an Outposts server a single interface is created for network connectivity. This default setting, depicted in the following diagram, is the most direct method for your instance to communicate externally.
Figure 1: Simple network connectivity on an Outposts server
When deploying an EC2 instance to an Outposts server, there are certain differences in using the default Elastic Network Interface (ENI) as compared to deploying in an AWS Region. Understanding these differences is critical before modifying the network configuration, which you do in the next step.
ENI differentiators between Outposts servers and the Region:
- Primary interface: The primary interface is an ENI. This ENI is associated to a subnet within a VPC. This VPC is extended from the Region to the Outposts server.
- IP address configuration: The primary network interface within the guest operating system (OS) of the EC2 instance must be configured to obtain an IP address through DHCP. The assigned IP address is from the IP address range of the VPC subnet associated with the Outposts server.
- Security group: A security group is associated with the ENI. This security group falls within the VPC that is extended from the Region. The user must apply appropriate access control rules to permit access to the EC2 instance. You may reuse security groups that already exist within the VPC.
- Outbound traffic: By default, an EC2 instance uses its ENI to direct outbound traffic toward the VPC subnet. Traffic flows according to the routing table associated with the Outposts server’s VPC subnet.
- Inbound traffic: If you’re only using an ENI, then traffic destined to EC2 instances on Outposts servers must traverse through the service link. In the preceding diagram, the user communicates with the EC2 instance over the internet. Traffic from the internet reaches the Region through the Internet Gateway of the VPC. Then, the VPC forwards the traffic to the appropriate subnet of the Outposts server (through the service link) and reaches the EC2 instance. The user must configure the necessary VPC components (Internet Gateway and associated routing table entries) for internet connectivity.
- Local network connectivity: There is no local network connectivity using the ENI. For local network connectivity, see the next section where we discuss the Outposts server Local Network Interface (LNI).
Local network connectivity for EC2 instances
Outposts servers allow you to communicate through the Local Network Interface (LNI) in addition to the ENI. The LNI is a logical networking component that connects the Amazon EC2 instances in your Outposts subnet to your on-premises network.
The Outposts server EC2 instance local communications characteristics:
- Local network traffic needs the use of an LNI.
- The subnets on Outposts servers must be enabled for LNIs. This is done by entering the following command:
aws ec2 modify-subnet-attribute \
--subnet-id subnet-1a2b3c4d \
--enable-lni-at-device-index 1
- IP address assignment for the LNI can be DHCP or static.
- You can’t apply VPC security groups to the LNI. To control traffic on the LNI, you can use an OS based firewall, external on-premises firewall, or other security devices.
- Amazon CloudWatch metrics are produced for each LNI.
- Outposts servers don’t tag VLAN traffic. If VLAN tags are needed, then the network interface settings inside the guest OS must apply the VLAN tags. Multiple VLAN interfaces can exist within the same LNI (in this case you would be using the LNI as a VLAN trunk).
- Local traffic bandwidth performance depends on the instance type. The larger the instance type, the higher performance the throughput of the LNI. The maximum throughput is 10 Gbps.
- EC2 instances that communicate locally always have at least two interfaces: one ENI and one or more LNIs. Therefore, the instance OS’s routing table must be configured based on the desired traffic behavior.
Example configuration: Local traffic for EC2 instance on Outposts server
Figure 2: Example scenario topology
In the example scenario, we want to launch an Amazon Linux 2023 instance and route all default traffic through the local network. Eth0 is the primary interface (ENI) and is used for traffic towards the Region. Eth1 is the LNI and is used for all other traffic. A user-data script is used to make the necessary routing changes at launch.
Here is a sample user-data script. These commands run as root so there is no need to prepend each command with sudo.
User data script (my_userdata.txt):
We can break down this script to observe the intent of each command:
When an instance is launched on Outposts server, the instance automatically has a default route that points toward the VPC through the ENI. In the example scenario, the desired configuration is to have all default traffic go through the LNI toward our on-premises LAN, not through the ENI. To accomplish this routing behavior for the ENI, we have to add a route toward the VPC and remove the default route. The first line adds a route through the VPC (172.31.0.0/16), using 172.31.239.1 as the gateway. The second line removes the default route that uses 172.31.239.1 (via the ENI) as the gateway.
Traffic not destined for the VPC routes through the LNI. This includes all local traffic and internet-bound traffic. The local network’s DHCP server provides a default-gateway in its DHCP lease. Therefore, there is already a default route assigned to the LNI. This steers any traffic without a more specific route, including internet traffic, toward the LNI.
Next, the user-data script makes the network settings persistent after reboot. The procedure varies depending on your OS. In the case of Amazon Linux 2023, it uses systemd-networkd.
cp -RL /run/systemd/network/* /etc/systemd/network/
This command copies the configuration files from the /run/systemd/network/ folder to /etc/systemd/network/. The configuration files in the /etc/systemd/network/ folder override the default settings and load during boot. The next is step is to modify the newly copied network configuration files.
echo -e '\n[Route] \nDestination=172.31.0.0/16 \nGateway=172.31.239.1 \nGatewayOnLink=yes' >> /etc/systemd/network/70-ens5.network
In this case the ENI is ens5. This line appends the static route section to the 70-ens5.network configuration file. This makes the static route added earlier in the script (route add -net 172.31.0.0/16 gw 172.31.239.1) persistent across reboots.
sed -i -e 's/UseGateway=true/UseGateway=false/g' /etc/systemd/network/70- ens5.network.d/eni.conf
Next, the user-script edits the configuration file, eni.conf, such that the default route isn’t used for the ENI at bootup. This is accomplished using sed to search and replace true with false for the UseGateway parameter.
Launching an instance with ENI and LNI
Now that the user-data script has been created, use the AWS Command Line Interface (AWS CLI) to launch an EC2 instance:
We can break down the parameters used in the preceding command:
--image-id ami-051f8a213df8bc089 \
This specifies the Amazon Machine Image (AMI) ID. ami-051f8a213df8bc089 is the AMI ID for Amazon Linux 2023 in us-east-1.
--count 1 \
This specifies how many EC2 instances to launch. You can launch multiple at the same time.
--instance-type c6id.xlarge
This specifies the instance type. By default, Outposts 2U servers are slotted with the c6id.8xlarge instance type and Outposts 1U servers are slotted with the c6gd.8xlarge instance type. You can adjust the slotting assignment during the ordering process or you can change the slotting assignment later by using the Self-service Capacity Management feature for AWS Outposts.
--key-name my_key
This specifies the public RSA key that is added to your EC2 instance. This key must already be defined in the same Region of your AWS account.
--user-data file://my_userdata.txt
This specifies the filename that contains your user-data script (that was created previously).
{ "DeviceIndex":0, "SubnetId":"subnet-0ca6abe6b34adfcce", "Groups": ["sg-0a9f8c2200c0a56f1"] }, \
{ "DeviceIndex":1, "SubnetId":"subnet-0ca6abe6b34adfcce", "Groups": ["sg-0a9f8c2200c0a56f1"] }]' \
This specifies the network interface configuration. By default, a single network interface, the ENI, is created. This example calls for a second interface for the LNI. DeviceIndex:0 is for the ENI and doesn’t change. DeviceIndex:1 is for the LNI, which we defined when we enabled LNI for the subnet (--enable-lni-at-device-index 1). The SubnetId refers to the subnet that was created on the Outposts server. If you want to deploy to a different Outposts server, then change the SubnetId. Groups refer to the security group that you would like assigned to the ENI. Security groups aren’t supported for the LNI, thus the security group specified for DeviceIndex:1 is only to comply with the command syntax check. A security group will not be applied to the LNI.
--tag-specifications '[{ "ResourceType":"instance","Tags":[ \
{ "Key":"Name", "Value":"server1" } ] }]'
This assigns a name to the EC2 instance, which in this case is server1.
Conclusion
AWS Outposts servers allow you to run native AWS services on-premises by providing local compute. This supports workloads with low latency and data residency requirements through on-premises processing.
Although Outposts servers integrate seamlessly with the AWS cloud, there are some unique networking considerations when deploying in your data center environment. Amazon EC2 instances on the Outposts server can route traffic over the AWS global network, but you can also enable Local Network Interfaces (LNIs) to directly access your on-premises networks.
In this post we’ve demonstrated using user-data scripts during instance launch to automate hybrid cloud networking flows tailored to your requirements. With proper planning, you can use the benefits of consistent AWS services and tooling while maintaining connectivity to your existing on-premises infrastructure.
Ready to get started with hybrid cloud networking on Outposts servers? Check out the Outposts server documentation and best practices guide to begin planning your on-premises deployment.