ENI configuration on RHEL and CentOS
EventBridge, Lambda, SSM Documents
Topics
Background knowledge: Symmetric/Asymmetric
Experimentations before: ping test
AWS solution: EventBridge, Lambda and SSM
Detail script walkthrough (optional)
Experimentations after: ping test
Reference
Background knowledge
Everyone who has configured two network interfaces to send and receive packets on a RHEL or CentOS distribution — version 7 and earlier — knows the hassle of setting them up.
When you want to attach a secondary network interface to an EC2 instance, one of the requirements from AWS is that both resources (ENI and EC2) are in the same subnet. This generally cause traffic flow issues between the first network interface — eth0 — and the second interface — eth1 — due to asymmetric routing.
Asymmetric routing. In simple words, it’s essentially when packets are received by one interface, and sent by another one. The graphic below gives a more visual representation about this networking mode, with traffic coming in through eth1, and leaving through eth0.
The reason behind this issue attains that both network interfaces — eth0 and eth1 — are in the same subnet. This means that they will share the same route table and gateway inside of the host. To overcome this issue, there needs to be a configuration in the server to change the routing mode.
Symmetric routing. It’s when one network interface sends and receives packets independently from the other one. The illustration below provides a more visual explanation on how this process works. In this case, eth1 is sending and receiving packets without necessarily being routed out through eth0, unlike the previous networking mode.
The problem now revolves around automating this process so that whenever attaching a new network interface onto an EC2 instance, this is taken care of without necessarily worrying about the configuration underneath.
Experiment before function
Below is an example of a secondary network interface eth1 (10.0.1.5) that was attached to a CentOS box. As you can see, this ENI has not been yet configured. When attempting to do a ping on the IP of this interface, packets are dropped because there is no route configuration on the host for this network interface.
AWS Solution
In this blog, I present to you a solution to automate this configuration in order to achieve a symmetric routing mode in the host. The general process for the solution works as follows:
EventBridge Rule
The first step is to set up ENI attachment detection through events. An event will be triggered every time a new ENI is attached to an EC2 instance. This event is captured through CloudTrail, which will record the API call for this attachment process. The payload looks similar to this:
The most important part of this is the InstanceId, which will be used at a later stage to determine which instance needs to be configured.
This is a .json configuration file to create an event that would be triggered once an ENI has been attached. Pay special attention to the EventPattern section. This is what determines the event call based on CloudTrail.
This is the command to set up the CloudWatch event.
aws event put-rule --cli-json-input file://your_path_to_file.jsonLambda
The second step is to create a lambda function which will be responsible for executing the shell script configuration through an SSM document. To achieve this, it is necessary to use the send_command() method from the SSM boto3 API. For this method to run, two parameters are required, the SSM Document name/version and the InstanceId.
This is the .json configuration file to create a basic lambda function. You can just fill in the values with your custom variables.
Here are the commands to deploy this function with AWS CLI.
aws lambda create-function --cli-input-json file://path_to_your_file.jsonSSM Document
Finally, the third step is to write an SSM document which contains a shell script that will configure the rules, routes and gateway inside of the host so that the second interface can receive and send traffic. Here is what the whole document looks like:
Here is the command to deploy this document using AWS CLI.
aws ssm create-document --content file://path_to_your_file.json --name "eni-attachment" --document-type "Command"Detail script walkthrough
This script has three fundamental components: configuring interface, editing IP rules and routes, and persisting changes in the host.
The first component involves:
- P1: Getting gateway ip address
- P2: Creating a ifcfg-ethX file inside of the /etc/sysconfig/network-scripts/ directory
- P3: Restarting the network
The second component involves:
- Getting the IPs from the first and second interfaces
- Adding IP rules and routes to the host network configuration based on the interface addr
The final component is:
- Creating new files, rule-ethX and route-ethX, in order for these changes to persist on host failure/reboot.
Experiment after function
After performing this setup, the same experimentation can be done once more. As seen in the picture below, this time the secondary network interface receives and sends packets, as opposed to the previous time where the packets were dropped.
Future improvements: shell script code needs to be more robust. Right now it has a lot of manual variables and hardcoded values.
Disclosure: function works pretty well in development environments, but it is currently lacking production testing.
Hope functions like this can help ease up management in your account.
Let me know if I can help in anything!
Go Build. 🔥
