Amazon VPC for On-Premises Network Engineers – Part 1

Editor’s note: This is the first of a popular two-part series by Nick Matthews. Read Part 2 >>

By Nick Matthews, Partner Solutions Architect, CCIE #23560

I relate better to networking when people use words and phrases I know, such as “router on a stick,” “MPLS WAN,” and “VRF,” and then show me a packet capture. That’s because I’m a network engineer, and it’s how we talk.

As more organizations move to Amazon Web Services (AWS), increasing numbers of network engineers are being asked to design and operate AWS networks. In this series of posts, I’ll explain Amazon Virtual Private Cloud (Amazon VPC) concepts using familiar dialect, to help network engineers at AWS Partner Network (APN) firms take advantage of networking in AWS.

If acronyms such as BGP, ECMP, ACL, VIP, and NIC are familiar to you, you’re in the right place. If not, you may want to look at the Amazon VPC documentation.

To start, Amazon VPC is the networking layer that AWS services use for connectivity. Amazon VPC was built to provide the scale, security, performance, and availability AWS customers require. Since few other networking environments have the same requirements as Amazon VPC, it’s hard to draw a simple comparison to existing networking technology.

To begin, I recommend watching Another Day, Another Billion Packets from re:Invent to understand what’s going on under the hood with packet forwarding, and how the data and control planes work.

The Fundamentals

Figure 1 – Virtual private cloud (VPC) example architecture.

Instance

This would be known as a virtual machine (VM) by most network engineers.

Availability Zone

An Availability Zone is a set of buildings, Internet uplinks, and power. You can think of it as a data center, but some Availability Zones contain more than one physical data center. There are currently 33 Availability Zones worldwide. An application deployed in a single Availability Zone is not considered highly available.

Region

An AWS Region is a collection of Availability Zones. You can think of a Region as containing multiple data centers within the same geographic area, with no shared fault domains. For example, a Region with 2 Availability Zones would be a good fit for disaster recovery with synchronous replication. AWS has 12 Regions across the world.

The majority of AWS services are regional, meaning that services can be run in multiple Availability Zones in the same AWS Region. Examples of Regions are Northern Virginia (us-east-1) and Northern California (us-west-1). You should design all traffic flows between AWS Regions as you would for Internet transit traffic, though in some cases flows will use AWS’s private backbone.

Virtual Private Cloud (VPC)

A VPC is a set of contained subnets with a common Classless Inter-Domain Routing (CIDR) block (up to a /16 netmask) running in a single geographic area (Region) across multiple data centers (Availability Zones).

A VPC is like a virtual data center, except that it’s physically spread out across Availability Zones. VPCs have network connectivity within the Region in which they are created. You can use Internet connectivity, virtual private network (VPN) connectivity, and VPC peering to connect VPCs to other networks. Address your VPC in RFC 1918 space so that it doesn’t conflict with other parts of your network.

When an account has multiple VPCs, they are similar to virtual routing and forwarding (VRF) tables or contexts on some devices. If you build a service in one VPC and need to access it from another VPC, you will either need to peer the two VPCs or use a VPN to connect them.

Instances in new accounts (created after December 4, 2013) are launched into VPCs by default. There is a default VPC that differs from VPCs created by users, as explained in the Amazon VPC documentation and explained below.

EC2-Classic Networking

Instances launched from accounts created before December 4, 2013 were on the EC2-Classic platform. Some accounts opened before that date can launch EC2-Classic instances today, but this blog post will not cover EC2-Classic networking.

There are many differences between EC2-VPC and EC2-Classic. For details, see the VPC documentation. With the exception of EC2-Classic instances, which live in a single network shared with other customers, all other instances live in a VPC.

Anatomy of a VPC

It’s useful to be handy with the building blocks of a VPC. The VPC wizard in the Amazon VPC console helps you get started, but network engineers will likely want to expand beyond the use cases defined in the VPC wizard.

Figure 2 – Basics of a VPC. A sample route table, network ACL, and security group are shown.

Default VPC

As I mentioned earlier, newer accounts (since December 4, 2013) include a default VPC, which mostly emulates the behavior of EC2-Classic networking for backward compatibility. Instances in the default VPC are given public addresses and a route to the Internet by default.

You can also create your own VPC. Instances in a user-created VPC do not receive a public IP address or a route to the Internet by default, but are configurable. Both default VPCs and user-defined VPCs can operate securely using features like security groups.

Subnet

Subnets take a variable portion of the CIDR block assigned to the VPC. Traditional subnetting rules and notation apply, so take care in subnet sizing, and design an address range that doesn’t overlap with on-premises data centers or other VPCs.

A subnet is specific to an Availability Zone, and there can be multiple subnets per Availability Zone. Each subnet has a single route table association, and each subnet is associated with a network access control list (network ACL). AWS uses five addresses in each subnet―the first four and the last address.

The minimum subnet size is a /28 netmask with 11 addresses. Subnets can be very large on AWS, since broadcast traffic is eliminated (see the “Routing and Switching” section later in this blog post for details). Also, devices like Elastic Load Balancer (ELB) use IP addresses, and many workloads are elastic and use more addresses as they scale up, so you should account for these when sizing subnets.

You may see references to public and private subnets. This is a shortcut for saying whether a subnet has a direct route to the Internet or not; otherwise, there are no differences. Typically, a public subnet has direct access to the Internet.

A private subnet has no external access―it connects through a virtual private gateway (VGW) to an on-premises data center, or uses Network Address Translation (NAT) for Internet-bound traffic. Subnets also have a setting called Auto-assign Public IP, which can give instances public IP addresses by default (and can be overridden per instance).

Route Table: Every subnet has a route table, and a single route table can be associated with multiple subnets. Route tables act like a source-based policy-based routing (PBR) rule. In other words, you can choose which direction packets should go based on the subnet the instance is in.

For example, private subnets might have route tables with a default route to a NAT gateway, and public subnets might have route tables with entries to the Internet using an Internet gateway. Private subnets can also have routes back to on-premises data centers using a virtual private gateway.

It’s tempting to describe route tables as being similar to virtual routing and forwarding (VRF) tables. However, subnets in the same VPC can communicate directly with one another, so the separation VRFs typically provide doesn’t apply. Functionally, everything in the same VPC has Layer 2 reachability. Use security groups and Network ACLs (explained below) to provide more granular access.

Route tables contain routes that point a CIDR block to an instance, Internet gateway, virtual private gateway, NAT gateway, VPC peer, or VPC endpoint. (More on these concepts later in this post.) Route tables can also receive dynamic BGP routes from a virtual private gateway.

A default route table is automatically attached to new subnets unless they’re manually associated with a different route table. For a higher level of control and flexibility, create a new route table for subnets instead of using the default route table. If a service is local to an Availability Zone (such as NAT Gateway), we recommend defining route tables per Availability Zone.

Security Group: A security group is like an access control list (ACL) typically applied to routers and firewalls, except that it is applied directly to the instance. Security groups handle the bulk of security in AWS to protect instances, and allow for fine-grained security per instance.

Security groups are stateful and track TCP, UDP, and ICMP connection status. Security groups contain security group rules, which are like ACL entries. Security group rules are whitelist only, and contain an implicit DENY ANY rule. Security groups can reference other security groups, acting similar to a network/object group in firewalls. This capability enables you to permit traffic from certain groups of instances without requiring the use of IP addresses. Overall, security groups are like a Layer 4 distributed firewall.

If you are interested in getting more detailed management of your security groups, check out APN Partners such as CloudCheckr and Dome9.

Network ACL: Network ACLs are applied per subnet, and provide a broader stroke than security groups. These would be used like a VLAN ACL (VACL) or Layer 2 ACL, though there is no Layer 2 in VPCs. Network ACLs are ordered and have PERMIT and DENY actions, with an implicit DENY ANY. Network ACLs, unlike security groups, are stateless. Also, network ACLs work only on CIDR ranges and can’t reference other objects such as instances. The default network ACL allows all packets.

If you want to block SSH for an entire subnet, you could add a DENY entry for TCP port 22 in the subnet’s network ACL. Otherwise, you would need to block SSH on each instance’s security group.

Elastic network interface (ENI): An elastic network interface (ENI) is like a virtual network interface card (NIC). You can apply multiple ENIs to an instance, and move the ENI to another instance in the same subnet. Multiple Elastic IP addresses can be applied to an ENI. An ENI has a dynamically assigned private address in the assigned subnet, and can optionally have a dynamically assigned public IP address as well. Multiple addresses can be assigned to an ENI.

Public IP address: A public IP address is a publicly routable address that is reachable from the Internet. There are two types of public IP addresses in a VPC: dynamically assigned addresses and Elastic IP addresses. By default, instances connected to the Internet receive a dynamically assigned public address. When the instance is terminated, the address is released. If you require that address to be persistent, you need an Elastic IP address. The public address will not change while the instance is operating.

Elastic IP address (EIP): An Elastic IP address is a static public IP address that is applied to an ENI, and can be associated to another instance after an instance is terminated. This makes it easier for things like ACL rules, DNS entries, and whitelisting in other systems. Every public address (whether it’s an Elastic IP address or not) has a private address associated with it. This private address is static unless the Elastic IP address is moved to another subnet. Also, remember that subnets are local to a single Availability Zone.

Cost of VPCs: VPCs, security groups, route tables, Internet gateways, virtual private gateways, network ACLs, and ENIs are free. VPN connections have an hourly charge. Elastic IP addresses have a cost if they are not associated with an instance, if an instance has more than one Elastic IP address, or if the instance is remapped more than 100 times in a month. Inbound traffic from the Internet is free, and outbound traffic has a data transfer fee. There is a data transfer fee between Availability Zones. For details, see Amazon VPC Pricing and Amazon EC2 Pricing on the AWS website.

External Connectivity

Let’s look at the external connectivity options. Each option is something you attach to your VPC, which makes it available to insert into route tables.

Figure 3 – External connectivity options for a VPC. This shows the architecture of a NAT gateway, VPN instance, internet gateway, VPN connections using a virtual private gateway, AWS Direct Connect connection, and VPC peering connection.

To connect an instance to the Internet in a non-default VPC, you will need to do the following:

1. Launch an instance.
2. Attach an Internet gateway to the VPC of the instance.
3. Add a 0.0.0.0/0 route pointed to the Internet gateway in the route table associated to the subnet of the instance.
4. Check that the security group and network access list allow appropriate ports and sources.

NAT gateway: A NAT gateway acts like an ECMP route to a set of NAT devices. These devices will do Network Address Translation (NAT) and Port Address Translation (PAT) for multiple hosts in a VPC to a single public Elastic IP address. (See Understanding NAT and PAT in Webopedia for an explanation of these terms.)

Previously, it was common for a single NAT instance, or a highly available NAT instance pair, to provide this function. A NAT gateway can perform NAT and PAT for less cost, with better performance, and less management overhead while providing better availability. To make this work, you point a default route to a NAT gateway attached to your VPC.

It’s critical to understand that a NAT gateway operates in a single Availability Zone. This differs from Internet gateways and virtual private gateways that are highly available across a region. This means that if you have services in multiple Availability Zones, you should have a NAT gateway in each zone.

This may affect your route tables, which should be assigned per subnet or Availability Zone. Traffic that flows between Availability Zones has a cost. For more information, see Amazon VPC Pricing.

Virtual private gateway (VGW): The virtual private gateway acts like an ECMP route to a set of WAN routers. It terminates two services: VPN and AWS Direct Connect. For VPN, the virtual private gateway terminates VPN tunnels and can propagate BGP routes into selected routing tables.

Direct Connect circuits will be pointed toward the virtual private gateway for connectivity into the VPC. Like the Internet gateway, a virtual private gateway is highly available without additional configuration. However, high availability for the VPN service and Direct Connect is configurable and managed by the user.

These two re:Invent presentations will help you design a high availability architecture for VPN and Direct Connect: AWS Direct Connect and VPNs and Double Redundancy with AWS Direct Connect.

Virtual private network (VPN): Virtual private networks (VPNs) in AWS come in two flavors: VPNs provided through the virtual private gateway, and instances running VPN software. In the case of the former, two tunnels are provided per customer gateway (CGW) for high availability.

Appropriate routing and failover must be configured for high availability between the tunnels. A virtual private gateway will not create a back-to-back VPN with another virtual private gateway, since both only respond to VPN connections. Routing works through Border Gateway Protocol (BGP) or static routing, with a maximum of 100 additional routes.

APN Partners such as FortyCloud and Ocedo can simplify creating VPN connections.

Customer gateway (CGW): A customer gateway is the on-premises firewall, router, or other device that terminates the VPN configuration. When you configure a customer gateway, sample configurations are available for many common device types.

For instances running VPN software, there are many APN Partner options (Check Point, Cisco, F5, Fortinet, Palo Alto, Sophos, to name a few) as well as open source options for VPN. These offer great flexibility, and high availability is achieved in numerous ways depending on the software. You can find more about these vendors on the AWS Security Partner Solutions page, and get the software from AWS Marketplace.

Virtual router: The VPC router is mentioned in the AWS documentation, but it’s a logical construct. It’s easier to read the following “Routing and Switching” section to understand traffic flow.

DNS in a VPC: By default, Domain Name System (DNS) is handled by the VPC. Each instance is assigned a DNS name with its address, which is automatically managed. It’s possible to use Amazon Route 53 to create your own DNS service inside a VPC with private hosted zones. You can alternatively use external DNS or host DNS in an EC2 instance.

Routing and switching: This is the fun part. The re:Invent session Another Day, Another Billion Packets, which I mentioned at the beginning of this blog post, covers this topic in depth.

There are a few deviations network engineers will want to know about routing and switching in a VPC:

• All traffic is unicast (the Address Resolution Protocol, or ARP, is gone too). The VPC forwarding design knows where all IP addresses are, and proxies ARP responses for all ARP requests locally at the hypervisor.
• All subnets in a VPC have direct access to one another, unless they are filtered in the host or in security groups or network ACLs.
• VPCs and VPC peering are local to a single region. Traffic between regions can connect using the Internet, VPNs, or Direct Connect.
• All traffic between two points (instances, virtual gateways, Internet gateways, or NAT gateways) in the same VPC is forwarded directly. Configuring a different default gateway, such as creating a firewall to be the default gateway between subnets, will not change this behavior. The VPC CIDR range is automatically added as a local route (up to a /16 netmask), and it can’t be overridden with a more specific route.
• If an instance will be acting as a proxy, router, firewall, or otherwise sending or receiving packets with a different IP address than the assigned ENI IP address, source/destination checking must be disabled. For more information, see the Amazon VPC documentation.
• There is no transitive routing. This can manifest itself in a few different ways.

Figure 4 – Without transitive routing, VPCs peered through a hub cannot reach one another―they can only reach directly connected VPCs. Full mesh peering would address this limitation.

• There’s a maximum transmission unit (MTU) of 1,500 when traffic leaves the VPC (between VPCs, regions, Direct Connect, or the Internet). Jumbo frames are supported within a VPC.
• There are placement groups that allow for full 10-gigabit Ethernet bandwidth connectivity between instances that have enhanced networking enabled. Enhanced networking is covered in the next part of this blog series.
• Instance types have different networking performance, as discussed on the Amazon EC2 Instance Types page. All traffic types are considered equal leaving an instance, because the Differentiated Services Code Point (DSCP) bits are not used for forwarding. Generally speaking, larger instance types will provide better network performance. If you have determined that bandwidth is a limitation, try enhanced networking, or a larger instance type or family.
• Instances should communicate using their private addresses, even when they have public addresses assigned. Using a public IP as a source will cause issues with the NAT that occurs. Furthermore, security groups won’t operate normally. Using a public IP instead of the private IP as a destination will route the traffic towards the Internet, which can cause suboptimal routing and data transfer charges. (See the Amazon VPC Pricing page.) The public IP assigned to an instance is effectively the public NAT address that would typically be assigned on a firewall or router.

Additional Resources

This wraps up the first post in this series where I’ve explained the fundamentals and anatomy of basic networking inside a VPC. In the next post I’ll explain how to create a VPC, and cover Direct Connect and some advanced VPC concepts. In the comments section, let us know what else you would like us to cover. We’re always interested in feedback from our APN partners and customers.

Read the second post in this series >>