How can I make my secondary network interface work in my CentOS or RHEL Amazon Elastic Compute Cloud (Amazon EC2) instance?

Adding a secondary network interface to a non-Amazon Linux instance causes traffic flow issues. The traffic flow issue differs depending on whether the secondary network interface is in the same subnet, or in a different subnet than the primary network interface.

When the primary and secondary network interfaces are in the same subnet

By default there is only one routing table with one gateway in the subnet. Traffic that comes into the secondary network interface tries to leave the instance using the primary network interface. This isn't allowed, because the secondary IP address doesn't belong to the Media Access Control (MAC) address of the primary network interface.

When the primary and the secondary network interfaces are in different subnets

Both the primary and the secondary network interfaces are in different subnets, and by default there is only one routing table. Only one of the network interfaces is used to manage non-local subnet traffic. Any non-local subnet traffic that comes into the network interface that isn't configured with the default gateway tries to leave the instance using the interface that has the default gateway. This isn't allowed, because the secondary IP address doesn't belong to the Media Access Control (MAC) address of the primary network interface.

Here is a summary of the steps to make the secondary interface work:

  1. Create a configuration file
  2. Create a new routing table
  3. Set rules in the Routing Policy Database
  4. Create a static route file

Note: For Ubuntu instances, see How can I make my secondary network interface work in my Ubuntu EC2 instance?

All procedures must be performed with root user privileges. Either become root with sudo -i, or execute all commands with sudo.

In the following example, the steps assume a 172.16.1.0/24 subnet, a 172.16.1.1 gateway, and secondary interface IPs ranging from 172.16.1.20, 172.16.1.30, and so on.

Note: This procedure was tested on a fresh AMI with no custom settings. If you have customizations there is no guarantee that this procedure will work. Always test the procedure on a development environment and back up your system before making any changes.

Create a secondary network configuration file

1.    Use grep to find the name of the primary network interface:  

ip a | grep ^[[:digit:]]

The output looks similar to the following:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc pfifo_fast state UP group default qlen 1000
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000

Important: In the previous example, the primary interfaces are named eth0, eth1 and so on. However, for instances that support enhanced networking, such as the m4 and m5 family type, you might see a naming inconsistency. For example, the primary might be named ens3 while the secondary is named eth0. This naming inconsistency happens when the secondary interface is added while the instance is running.

You can avoid a naming inconsistency by adding the interface at launch time, or rebooting the instance. Or, while the interface is running, you can change the name using the ip link command:

ip link set eth0 name ens4 && ip link set ens4 up

This example changes the interface name eth0 to ens4 to align it with the primary interface named ens3.

2.    Open the primary network interface ifcfg-eth0 configuration file:

vi /etc/sysconfig/network-scripts/ifcfg-eth0

3.    Edit the primary network interface file so that it is similar to the following. Adapt the edits to your scenario:

DEVICE=eth0
NAME=eth0
HWADDR=00:00:00:00:00:00
BOOTPROTO=dhcp
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
NM_CONTROLLED=no

4.    Create or edit the secondary network interface ifcfg-eth1 configuration file:

vi /etc/sysconfig/network-scripts/ifcfg-eth1

5.    Edit the secondary interface file so that the file contents are similar to the following. Adapt the edits to your scenario, including your secondary interface MAC address. You can find the secondary interface MAC address using the ip addr command.

DEVICE=eth1
NAME=eth1
HWADDR=00:00:00:00:00:00
BOOTPROTO=dhcp
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
NM_CONTROLLED=no

If you have more than one IP on your secondary interface, add all the IP addresses manually to the ifcfg-eth1 file. The following example shows multiple IP addresses added to the file:

DEVICE=eth1
NAME=eth1
HWADDR=00:00:00:00:00:00
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
NM_CONTROLLED=no
DEFROUTE=no
IPADDR=172.16.1.20
PREFIX=24
IPADDR1=172.16.1.30
PREFIX1=24
IPADDR2=172.16.1.40
PREFIX2=24

6.    Edit the /etc/sysconfig/network file to be sure that the default gateway remains on the main routing table so that the primary interface doesn't lose connectivity:

vi /etc/sysconfig/network

Now add eth0 as the gateway device:

GATEWAYDEV=eth0

7.    To prevent the cloud-init from resetting your custom network configurations, add the following lines to the /etc/cloud/cloud.cfg file:

network:
 ; config: disabled

8.    By default, Red Hat Linux versions 7 and later come with NetworkManager enabled. But NetworkManager can interfere with network settings, especially the routing table. Stop the NetworkManager and disable it from starting again:

systemctl stop NetworkManager
systemctl disable NetworkManager

The systemctl method of disabling works only on systems that run systemd. On older systems, such as RHEL 6.5 or 6.8, you can disable NetworkManager by using the sysV and chkconfig command:

service NetworkManager stop
chkconfig NetworkManager off

9.    Restart the network:

systemctl restart network

In some cases, the network restart might fail if a different device naming is used along with the ifcfg-eth0 file. If the restart fails, delete the ifcfg-eth0 file, kill the dhcp client so that it stops adding stale route entries, and then restart the network:

rm -rf /etc/sysconfig/network-scripts/ifcfg-eth0
kill "ps -ef |grep dhclient|grep -v grep|awk '{print $2}'"
systemctl restart network

Create a new secondary routing table

The main routing table has an ID of 254 on Linux. By default, interface traffic is routed based on this table. You must create a new routing table for the secondary interface so that the traffic that arrives on the secondary interface can leave from the same interface.

1.    Find and take note of your default gateway:

ip route | grep default

2.    Create a new routing table for the secondary interface, and then add the default gateway route using the IP of the gateway you found on the previous step. In this example, the new table is ID 1000, and the IP is 172.16.1.1:

ip route add default via 172.16.1.1 dev eth1 table 1000

3.    Be sure that table 1000 has a route for every IP present on the secondary interface.

The following is an example for two IPs on a secondary network interface:

ip route add 172.16.1.20 dev eth1 table 1000
ip route add 172.16.1.30 dev eth1 table 1000

4.    Review table 1000 and be sure that the routes you added in the previous step are visible. The following ip route show command displays the default route and the routes for each IP on the secondary interface:

ip route show table 1000

Set rules in the routing policy database

1.    Use the ip rule add command to set rules for every IP present on the secondary interface in the routing policy database. This makes sure that traffic coming from these IPs is routed according to table 1000.

ip rule add from 172.16.1.20 lookup 1000
ip rule add from 172.16.1.30 lookup 1000

2.    Ping the private IP or the corresponding Elastic IP address to validate the connectivity to the IPs on the secondary network interface. If you have Elastic IPs that are pointing to these IPs, then they can be accessed from the public network as well. The security groups attached to the secondary network interface should allow ICMP traffic for the ping test to return a successful response.

The following example shows a three-count ping to eth0 IP and to a newly added eth1 IP from another instance within the same VPC:

ping -c3 172.16.1.10
ping -c3 172.16.1.20

Create a secondary static route file

1.    Create the route-eth1 static route file:

vi /etc/sysconfig/network-scripts/route-eth1

2.    Enter the same routes as you entered when creating the secondary routing table at the command line:

default via 172.16.1.1 dev eth1 table 1000
172.16.1.20 dev eth1 table 1000
172.16.1.30 dev eth1 table 1000

3.    Create or edit a rule file for rule-eth1:

vi /etc/sysconfig/network-scripts/rule-eth1

Put in the same routes as you entered on the command line when creating the new secondary routing table:

from 172.16.1.20 lookup 1000
from 172.16.1.30 lookup 1000

After the static routes and rules files are in place, routing and rules will persist across instance stops and starts, or when rebooting the interface.


Did this page help you? Yes | No

Back to the AWS Support Knowledge Center

Need help? Visit the AWS Support Center

Published: 2018-11-09

Updated: 2019-02-28