Containers

API Gateway as an Ingress Controller for Amazon EKS

When teams deploy microservices on Amazon EKS, they usually expose a REST API for use in front ends and third-party applications. A best practice is to manage these APIs with an API Gateway.

This provides a unique entry point for your APIs and also eliminates the need to implement API-specific code for things like security, caching, throttling, and monitoring for each of your microservices. You can implement this pattern using ALB Ingress Controller and Amazon API Gateway. Amazon API Gateway is a fully managed service for managing secure APIs at any scale. This approach works, but it requires you to create several configuration files. How can we automate this task?

In this post, we show you how to use an open-source solution API Gateway Ingress Controller, which reduces the manual steps by quickly configuring your APIs running on Amazon EKS by leveraging the HTTP proxy mode of Amazon API Gateway. API Gateway Ingress Controller configures a Network Load Balancer in front of a reverse proxy pod, which handles the path-based routing and routes HTTP requests to the pods. The following diagram shows the high-level architecture described in this article:

How Kubernetes Ingress works with Amazon API Gateway Ingress Controller

The following diagram details the AWS components created by Amazon API Gateway Ingress Controller when a user creates an Ingress resource. The Ingress resource routes Ingress traffic from the API Gateway to the Kubernetes cluster using a Private Network Load Balancer via API Gateway VPC Link.

Ingress creation

Following the steps in the numbered blue circles in the above diagram:

  1. The API Gateway Ingress Controller watches for Ingress events from the API server. When it finds Ingress resources that satisfy its requirements, it begins the creation of AWS resources.
  2. API Gateway API is created, and the specified API Gateway stages outlined in Ingress annotations are created.
  3. A Private Network Load Balancer is created for the Ingress resource, and Listeners are created for every port specified in paths configuration.
  4. For the API Gateway to communicate to the Private Network Load Balancer, a Private API Gateway VPC Link is created.
  5. TargetGroup is created for the reverse proxies specified in the Ingress resource.
  6. The API Gateway Ingress Controller deploys NGINX as the reverse proxy, and the reverse proxy Rules are created for each path specified in your Ingress resource. This ensures that traffic to a specific path is routed to the correct pod.

Step 1: Create AWS Cloud9 IDE

We recommend using AWS Cloud9 IDE as it was used to create the directions for this blog.

Create an AWS Cloud9 environment

Create it in the Oregon Region by following this deep-link 

  • Name it “eksworkshop,” click Next.
  • Choose “t2.small” for instance type, take all default values, and click Create an environment
  • When it comes up, customize the environment by closing the welcome tab and lower work area, and opening a new terminal tab in the main work area:

Create an IAM role for your workspace

  • Follow this deep-link to create an IAM role with AdministratorAccess.
  • Confirm that AWS service and EC2 are selected, then click Next to view permissions.
  • Confirm that AdministratorAccess is checked, then click Next: Tags to assign tags.
  • Take the defaults, and click Next: Review to review.
  • Enter “eksworkshop-admin” for the name, and click Create role.

 

Attach the IAM role to your workspace

  • Follow this deep-link to find your AWS Cloud9 EC2 instance
  • Select the instance, then choose Actions / Instance Settings / Attach/Replace IAM role
  • Choose eksworkshop-admin from the IAM role drop down, and select Apply

Update workspace IAM settings

  • Click on the clog in the top-right corner of the workspaces
  • Select AWS SETTINGS
  • Turn off AWS managed temporary credentials
  • Close the Preferences tab

To ensure temporary credentials aren’t already in place, we will also remove any existing credentials file:

rm -vf ${HOME}/.aws/credentials

 

Use the GetCallerIdentity CLI command to validate that the AWS Cloud9 IDE is using the correct IAM role.

aws sts get-caller-identity
#The output assumed-role name should contain
eksworkshop-admin

 

Install kubectl and other tools

More information for kubectl installation, see Installing kubectl.

sudo curl --silent --location -o /usr/local/bin/kubectl \
 https://amazon-eks.s3-us-west-2.amazonaws.com/1.14.6/2019-08-22/bin/linux/amd64/kubectl
 sudo chmod +x /usr/local/bin/kubectl

Install jq, envsubst (from GNU gettext utilities) and bash-completion

sudo yum -y install jq gettext bash-completion

Download and install the eksctl binary

curl --silent --location \
 "https://github.com/weaveworks/eksctl/releases/download/latest_release/eksctl_$(uname -s)_amd64.tar.gz" \
  | tar xz -C /tmp
  
sudo mv -v /tmp/eksctl /usr/local/bin

Confirm the eksctl command works

eksctl version

Clone the GitHub project for the blog. It has all the YAML files needed for this blog.

cd ~/environment/
git clone https://github.com/aws-samples/amazon-apigateway-ingress-controller-blog.git

Step 2: Create an Amazon EKS cluster

We will use eksctl to launch and configure our Amazon EKS cluster and nodes.

eksctl create cluster --name=eksworkshop-blog  --nodes=3 --region=us-west-2

It will take up to 15 minutes to create the Amazon EKS cluster. Once the cluster is running, and you have access to it by running the following command.

kubectl get nodes 
# if we see our 3 nodes, we know we have authenticated correctly, cluster is up

 

Step 3: Setup kube2iam role

Amazon API Gateway Ingress Controller needs four policies to create the AWS resource, and those are AutoScallingFullAccess, AmazonAPIGatewayAdministrator, AmazonVPCFullAccess, and AWSCloudFormationFullAccess. One option is to add these policies to node’s instance role, which is not recommended. IAM roles for Service Accounts is the best option. The API Gateway Ingress Controller currently does not support this option. It is on the to-do list for the project. We will set up kube2iam to assign a role to the Ingress Controller pod, let’s create this role.

 

Fetch node role

We need the following three commands to get the role name attached to the cluster’s nodes.

STACK_NAME=$(eksctl get nodegroup --cluster eksworkshop-blog -o json | jq -r '.[].StackName')
ROLE_NAME=$(aws cloudformation describe-stack-resources --stack-name $STACK_NAME | jq -r '.StackResources[] | select(.ResourceType=="AWS::IAM::Role") | .PhysicalResourceId')
aws iam get-role --role-name $ROLE_NAME | jq -r  .Role.Arn

#It will display node's Role Arn, copy it Edit ~/environment/apigw-ingress-controller-blog/kube2iam-ingress-trust-policy.yml and replace the role ARN.
.....
"Principal": {
"AWS": "arn:aws:iam::xx:role/eksctl-eksworkshop-blog-nodegroup-NodeInstanceRole-xxxxxx"
},
......

Create Kube2Iam role

Create a role named kub2iam-ingress-role, and the name is important as its configured in ~/environment/apigw-ingress-controller-blog/AmazonAPIGWHelmChart/template/statefulset.yaml

cd ~/environment/apigw-ingress-controller-blog

aws iam create-role --role-name kube2iam-ingress-role \
 --assume-role-policy-document file://kube2iam-ingress-trust-policy.yml

We need to add four policies to the kub2iam-ingress-role. The Amazon API Gateway Ingress Controller will use it to create AWS resources.

aws iam attach-role-policy --role-name kube2iam-ingress-role --policy-arn \
 arn:aws:iam::aws:policy/AutoScalingFullAccess  
aws iam attach-role-policy --role-name kube2iam-ingress-role --policy-arn \
 arn:aws:iam::aws:policy/AmazonAPIGatewayAdministrator
aws iam attach-role-policy --role-name kube2iam-ingress-role --policy-arn \
 arn:aws:iam::aws:policy/AmazonVPCFullAccess 
aws iam attach-role-policy --role-name kube2iam-ingress-role --policy-arn \
 arn:aws:iam::aws:policy/AWSCloudFormationFullAccess 
aws iam attach-role-policy --role-name kube2iam-ingress-role --policy-arn \
 arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess
 

Step 4: Setup Helm and deploy charts

Currently, the API Gateway Ingress Controller is available as a Helm chart. Follow these steps to install Helm.

Install Helm CLI

cd ~/environment
curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > get_helm.sh
chmod +x get_helm.sh
./get_helm.sh

Configure Helm access with RBAC

cd ~/environment/apigw-ingress-controller-blog
kubectl apply -f helm-rbac.yml
helm init --service-account tiller

Deploy kube2iam Chart

Install the kube2iam Helm chart.

helm install \
  --set rbac.create=true \
  --set host.iptables=true \
  --set host.interface=eni+ \
  --set=extraArgs.auto-discover-base-arn= \
  stable/kube2iam

Verify the kube2iam pod is installed and running.

kubectl get pods
#It will list out three pods with kube2iam in their name

#View the logs of one of the pods, replace pod name  
kubectl logs xxxx-kube2iam-xxx

#Logs should contain "base ARN autodetected"
time="2019-12-27T23:08:26Z" level=info msg="base ARN autodetected, arn:aws:iam::xx:role/"

 

Deploy API Gateway Ingress Controller Chart

Install the API Ingress Controller Helm chart.

cd ~/environment/apigw-ingress-controller-blog/AmazonAPIGWHelmChart

helm install --debug ./amazon-apigateway-ingress-controller \
 --set image.repository="karthikk296d/aws-apigw-ingress-controller"

Verify that the API Gateway Ingress Controller is installed run the following command.

kubectl get pods
#You will see one pod for apigw Ingress controller
NAME                                            READY   STATUS             RESTARTS   AGE
virtuous-markhor-amzn-apigw-ingress-controller-0   1/1     Running   0          22s          26s

 

Step 5: Deploy the sample REST API

Deploy Book and Author – sample microservices for this blog.

cd ~/environment/apigw-ingress-controller-blog
kubectl apply -f book-deployment.yml
kubectl apply -f book-service.yml 
kubectl apply -f author-deployment.yml
kubectl apply -f author-service.yml 

What type of service are the book and the author? Let’s take a peek at the book-service.yml. Book service and Author Service are both of type ClusterIP. ClusterIP can only be invoked from within the Amazon EKS cluster.

......
  name: bookservice
spec:
  type: ClusterIP
  selector:
    app: books
....

You will see the following pod running on the cluster now.

kubectl get pods
# You will see author and book pod running
NAME                                            READY   STATUS    RESTARTS   AGE
author-deployment-658d67f7dc-9z4r6              1/1     Running   0          14m
book-deployment-577b7577ff-pn5ng 

And you will see the following services running.

kubectl get svc
# You will see authorservice and booksservice running
NAME               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
authorservice      ClusterIP   10.100.10.144    <none>        80/TCP    15m
bookservice        ClusterIP   10.100.128.1     <none>        80/TCP    15m

Step 6: Deploy API Gateway Ingress Controller resources

API Gateway Ingress currently secures the REST API with IAM authentication. Let’s create a user we will use to invoke the sample REST API.

aws iam create-user --user-name apigw-user
# an user named apigw-user will be created

Replace apigateway.ingress.kubernetes.io/client-arns default user ARN in ~/environment/apigw-ingress-controller-blog/api_ingress.yml with the apigw-user’s Arn.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    *#Replace arn::foo,arn::bar with a apigw-user's Arn*
    apigateway.ingress.kubernetes.io/client-arns: arn:aws:iam::xxxxx:user/apigw-user
    apigateway.ingress.kubernetes.io/stage-name: prod
    kubernetes.io/ingress.class: apigateway
  name: api-95d8427d
  namespace: default
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: bookservice
          servicePort: 80
        path: /api/book
      - backend:
          serviceName: authorservice
          servicePort: 80
        path: /api/author

After editing the api_ingress.yml file, deploy it by running the following command.

cd ~/environment/apigw-ingress-controller-blog/
kubectl apply -f api_ingress.yml

Verify the Ingress is up and running.

kubectl get ingress
#You will see one ingress deployed named api-95d8427d  

Verify the reverse proxies deployment.

kubectl get pods
#You will see three reverse proxy pods running
NAME                                               READY   STATUS    RESTARTS   AGE
api-95d8427d-reverse-proxy-5fc845fc67-bx7t2        1/1     Running   0          4s
api-95d8427d-reverse-proxy-5fc845fc67-d7gxf        1/1     Running   0          4s
api-95d8427d-reverse-proxy-5fc845fc67-mttkm        1/1     Running   0          4s

To see the progress of the Ingress resources creation view the logs of the xxxx-amzn-apigw-ingress-controller pod, you can also view the progress using AWS Management Console CloudFormation.

kubectl logs xxxx-markhor-amzn-apigw-ingress-controller-0 -f

# you will see create in progress, message
"Not complete, requeuing","status":"CREATE_IN_PROGRESS"}

#All the resources will be created in around 15 minutes
{"level":"info",........,"msg":"Stack Create/Update Complete"}

 

Step 7: Test the deployment

Verify api-95d8427d creation in API Gateway Console.

Click on api-95d8427d, and let’s review its configuration.

Click on resource VPC Link in the left menu, and this was created by the see Amazon API Gateway Ingress Controller. It provides access to HTTP(S) resources within your Amazon Virtual Private Cloud (VPC) without exposing them directly to the public internet.

Click on resource policy left menu, Amazon API Gateway Ingress Controller configured access control to this private API using a resource policy.

 

Click on the stages link in the left menu, and you see the prod stage. Click on /api/author, and then method get. You see the invoke URL for the API, copy it.

Let’s use the apigw-user’s access key and secret key to invoke the API. You can get the credentials with this command.

aws iam create-access-key --user-name apigw-user

#From the output copy the access key and secret key, 
#so you can use it to invoke the API
{
    "AccessKey": {
        "UserName": "apigw-user", 
        "Status": "Active", 
        "CreateDate": "2019-12-26T22:11:34Z", 
        "SecretAccessKey": "xxxxx", 
        "AccessKeyId": "xxxxx"
    }
}

It is a better practice to use temporary credentials when invoking APIs. You can use aws sts get-session-token command to get the temporary credentials. You can find more information on how to use this command here.

Use the Postman to invoke the REST API, paste the copied API URL, and make sure the URL ending has the following /api/author/list.

Click on the Authorization tab and provide access key and secret key and optionally Session token and click on Send.

 

You will see a list of authors as the output of the request. You can try following API Urls and notice different outputs.

#Change the API Url to following to get a list of Books
https://xxxxxxx.execute-api.us-east-1.amazonaws.com/prod/api/book/list

#Copy one of the ISBN13 from the out and change the api URL to following 
https://xxxxxxx.execute-api.us-east-1.amazonaws.com/prod/api/book/9781617293825        

 

Clean up

To clean up, let’s delete the deployments and services.

cd ~/environment/api-gateway-ingress-blog/
kubectl delete -f book-deployment.yml
kubectl delete -f book-service.yml 
kubectl delete -f author-deployment.yml
kubectl delete -f author-service.yml 

When you delete the API Gateway Ingress Controller, it removes the Network Load Balancer, API Gateway API, and other resources it has created.

kubectl delete ingress api-95d8427d

Delete the Helm chart for the API Gateway Ingress Controller

# get the name of your API Gateway ingress Chart with the following command
helm ls

# The name of your chart will be different 
NAME            REVISION      STATUS          CHART                                           APP VERSION     NAMESPACE
lame-fox        1             DEPLOYED        amazon-apigateway-ingress-controller-0.1.0      1.0             default
NAME                    CHART                                           
filled-butterfly        amazon-apigateway-ingress-controller-0.1.0   
newbie-narwhal          kube2iam-2.1.0        

# Issue the following command to delete both Helm charts
helm delete filled-butterfly
helm delete newbie-narwhal

If you don’t need Amazon EKS cluster, you can delete it.

eksctl delete cluster --name=eksworkshop-blog

Get involved

The Amazon API Gateway Ingress Controller is a fully open-source project maintained by Karthikk Dhandapani and Anand Modh. Karthikk and Anand are System Development Engineers at Amazon. They are a customer of the Amazon EKS inside Amazon Fulfillment Technologies and are obsessed with automation. They like to build containerized and Serverless Applications. The AWS team has also tested the Ingress Controller with Amazon EKS, and it currently supports Kubernetes version 1.14.

 

More resources:

 

Haider Naqvi

Haider Naqvi

Haider Naqvi is a Solutions Architect at Amazon Web Services. He has extensive Software Development and Enterprise Architecture experience. He is based out of New York.

Re Alvarez-Parmar

Re Alvarez-Parmar

Re Alvarez-Parmar is a Container Specialist Solutions Architect at Amazon Web Services. He helps customers use AWS container services to design scalable and secure applications. He is based out of New York and uses Twitter, sparingly, @realz