Containers

Protecting your Amazon EKS web apps with AWS WAF

Analyze the traffic patterns on any public-facing website or web app, and you’ll notice connection requests from all over the world. Apart from the intended traffic, a typical web application responds to requests from bots, health checks, and various attempts to circumvent security and gain unauthorized access.

In addition to impacting your customer’s experience, these requests can also increase your AWS spend. Today, business-critical web apps rely on web application firewalls to block unwanted traffic and protect apps from common vulnerabilities. But many customers struggle with the complexity when it comes to implementing an effective web application firewall. AWS Web Application Firewall (AWS WAF) and AWS Firewall Manager are designed to make it easy for you to protect your web applications and APIs from common web exploits that can disrupt services, increase resource usage, and put data at risk.

This post describes how to use AWS WAF and AWS Firewall Manager to protect web-based workloads that run in an Amazon Elastic Kubernetes Services (Amazon EKS) cluster.

AWS WAF gives you control over the type of traffic that reaches your web applications. It allows you to monitor HTTP(S) requests to web applications and protect them against DDoS attacks, bots, and common attack patterns, such as SQL injection or cross-site scripting.

If you are unfamiliar with web application firewalls (WAF), you’d be pleased to learn that you don’t have to be a regular expressions wrangler or an expert in firewall rules to use AWS WAF. It comes with a set of AWS-managed rules, so you don’t have to write filtering logic to protect against common application vulnerabilities or unwanted traffic.

AWS WAF integrates with Amazon CloudFront, Application Load Balancer (ALB), Amazon API Gateway, and AWS AppSync. If you already use an ALB as an ingress for your Kubernetes-hosted applications, you can add a web application firewall to your apps within a few minutes.

Customers that operate in multiple AWS accounts can use AWS Organizations and AWS Firewall Manager to control AWS WAF rules in multiple accounts from a single place. AWS Firewall Manager monitors for new resources or accounts created to ensure they comply with a mandatory set of security policies. It is a best practice to run EKS clusters in dedicated VPCs, and Firewall Manager can ensure your WAF rules get applied across accounts, wherever your applications run.

Solution

This post demonstrates how to implement a web application firewall using AWS WAF to protect applications running on EKS. We will start by creating an EKS cluster and deploying a sample workload. The sample application that we will use for this walkthrough is a web-based application that we’ll expose using an Application Load Balancer. We’ll then create a Kubernetes ingress and associate an AWS WAF web access control list (web ACL) with an ALB in front of the ingress. solution architecture diagram

Prerequisites

You will need the following to complete the tutorial:

Create an EKS cluster

Let’s start by setting a few environment variables:

WAF_AWS_REGION=us-west-2 <-- Change this to match your region
WAF_ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text)
WAF_EKS_CLUSTER_NAME=wa-eks-sample

Create a cluster using eksctl:

eksctl create cluster \
  --name $WAF_EKS_CLUSTER_NAME \
  --region $WAF_AWS_REGION \
  --managed

Store the cluster’s VPC ID in an environment variable as we will need it for the next step:

WAF_VPC_ID=$(aws eks describe-cluster \
  --name $WAF_EKS_CLUSTER_NAME \
  --region $WAF_AWS_REGION \
  --query 'cluster.resourcesVpcConfig.vpcId' \
  --output text)

Install the AWS Load Balancer Controller

The AWS Load Balancer Controller is a Kubernetes controller that runs in your EKS cluster and handles the configuration of the Network Load Balancers and Application Load Balancers on your behalf. It allows you to configure Load Balancers declaratively in the same manner as you handle the configuration of your application.

Install the AWS Load Balancer Controller by running these commands:

## Associate OIDC provider 
eksctl utils associate-iam-oidc-provider \
  --cluster $WAF_EKS_CLUSTER_NAME \
  --region $WAF_AWS_REGION \
  --approve
 
## Download the IAM policy document
curl -S https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.2.0/docs/install/iam_policy.json -o iam-policy.json 
## Create an IAM policy
WAF_LBC_IAM_POLICY_ARN=$(aws iam create-policy \
  --policy-name AWSLoadBalancerControllerIAMPolicy-WAFDEMO \
  --policy-document file://iam-policy.json \
  --query 'Policy.Arn' \
  --output text)
## Create a service account 
eksctl create iamserviceaccount \
  --cluster=$WAF_EKS_CLUSTER_NAME \
  --region $WAF_AWS_REGION \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --override-existing-serviceaccounts \
  --attach-policy-arn=arn:aws:iam::${WAF_ACCOUNT_ID}:policy/AWSLoadBalancerControllerIAMPolicy-WAFDEMO \
  --approve
  
helm repo add eks https://aws.github.io/eks-charts && helm repo update
kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=master"
helm install aws-load-balancer-controller \
  eks/aws-load-balancer-controller \
  --namespace kube-system \
  --set clusterName=$WAF_EKS_CLUSTER_NAME \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --set vpcId=$WAF_VPC_ID \
  --set region=$WAF_AWS_REGION

Deploy the sample app

We will use a sample application called Yelb for this demo. It provides an Angular 2-based UI that will represent a real-world application for this post. Here’s a high-level architectural view of Yelb:high-level architectural view of Yelb:

Clone the repository and deploy Yelb in your EKS cluster:

git clone https://github.com/aws/aws-app-mesh-examples.git
cd aws-app-mesh-examples/walkthroughs/eks-getting-started/
kubectl apply -f infrastructure/yelb_initial_deployment.yaml

The Postgres database that Yelb uses is not configured to use a persistent volume.

Expose Yelb using an ingress

Let’s create a Kubernetes ingress to make Yelb available publicly. The AWS Load Balancer Controller will associate the ingress with an Application Load Balancer.

cat << EOF > yelb-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: yelb.app
  namespace: yelb
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: yelb-ui
                port:
                  number: 80
EOF
kubectl apply -f yelb-ingress.yaml  

Test the application by sending a request using curl or by using a  web browser to navigate to the URL. You can obtain the URL using Kubernetes API:

 kubectl get ingress yelb.app -n yelb \
  -o jsonpath="{.status.loadBalancer.ingress[].hostname}")

If you receive an error, retry after a few minutes. DNS propagation can take a few minutes depending on your network setup.

Add a web application firewall to the ingress

Now that our sample application is functional, let’s add a web application firewall to it. The first thing we need to do is create a WAS web ACL. In AWS WAF, a web access control list or a web ACL monitors HTTP(S) requests for one or more AWS resources. These resources can be an Amazon API Gateway, AWS AppSync, Amazon CloudFront, or an Application Load Balancer.

Within an AWS WAF Web ACL, you associate rule groups that define the attack patterns to look for in web requests and the action to take when a request matches the patterns. Rule groups are reusable collections of rules. You can use Managed rule groups offered and maintained by AWS and AWS Marketplace sellers. When you use managed rules, AWS WAF automatically updates your WAF Rules regularly to ensure that your web apps are protected against newer threats. You can also write your own rules and use your own rule groups.

Create an AWS WAF web ACL:

WAF_WACL_ARN=$(aws wafv2 create-web-acl \
  --name WAF-FOR-YELB \
  --region $WAF_AWS_REGION \
  --default-action Allow={} \
  --scope REGIONAL \
  --visibility-config SampledRequestsEnabled=true,CloudWatchMetricsEnabled=true,MetricName=YelbWAFAclMetrics \
  --description "WAF Web ACL for Yelb" \
  --query 'Summary.ARN' \
  --output text )

Store the AWS WAF web ACL’s Id in an environment variable as it is required for updating the AWS WAF web ACL in the upcoming steps:

WAF_WAF_ID=$(aws wafv2 list-web-acls \
  --region $WAF_AWS_REGION \
  --scope REGIONAL \
  --query "WebACLs[?Name=='WAF-for-Yelb'].Id" \
  --output text)

Update the ingress and associate this AWS WAF web ACL with the ALB that the ingress uses:

cat << EOF > yelb-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: yelb.app
  namespace: yelb
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/wafv2-acl-arn: ${WAF_WACL_ARN}
spec:
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: yelb-ui
                port:
                  number: 80
EOF
kubectl apply -f yelb-ingress.yaml  

By adding alb.ingress.kubernetes.io/wafv2-acl-arn annotation to the ingress, AWS WAF is inspecting incoming traffic. However, it’s not blocking any traffic yet. Let’s send a request to our sample app using curl. Note that our request is handled as intended:

curl $(kubectl get ingress yelb.app -n yelb \
  -o jsonpath="{.status.loadBalancer.ingress[].hostname}")

You should see a response from Yelb’s UI server:

response from Yelb's UI server

Enable traffic filtering in AWS WAF

We have associated the ALB that our Kubernetes ingress uses with an AWS WAF web ACL Every request that’s handled by our sample application Yelb pods goes through AWS WAF for inspection. The AWS WAF web ACL is currently allowing every request to pass because we haven’t configured any AWS WAF rules. In order to filter out potentially malicious traffic, we have to specify rules. These rules will tell AWS WAF how to inspect web requests and what to do when it finds a request that matches the inspection criteria.

AWS WAF Bot Control is a managed rule group that provides visibility and control over common and pervasive bot traffic to web applications. The Bot Control managed rule group has been tuned to detect various types of bots seen on the web. It can also detect requests that are generated from HTTP libraries, such as libcurl.

Since our sample workload isn’t popular enough to attract malicious traffic, let’s use curl to generate bot-like traffic. Once enabled, we expect users who are accessing our application from a web browser like Firefox or Chrome to be allowed in, whereas traffic generated from curl would be blocked out.

While Bot Control has been optimized to minimize false positives, we recommend that you deploy Bot Control in count mode first and review CloudWatch metrics and AWS WAF logs to ensure that you are not accidentally blocking legitimate traffic. You can use the Labels feature within AWS WAF to customize how Bot Control behaves. Based on labels generated by Bot Control, you can have AWS WAF take an alternative action, such as sending out customized responses back to the client. Customers use custom responses to override the default response, which is 403 (Forbidden), for block actions when they’d like to send a nondefault status, serve a static error page code back to the client, or redirect the client to a different URL by specifying a 3xx redirection status code.

Create a rules file:

cat << EOF > waf-rules.json 
[
    {
      "Name": "AWS-AWSManagedRulesBotControlRuleSet",
      "Priority": 0,
      "Statement": {
        "ManagedRuleGroupStatement": {
          "VendorName": "AWS",
          "Name": "AWSManagedRulesBotControlRuleSet"
        }
      },
      "OverrideAction": {
        "None": {}
      },
      "VisibilityConfig": {
        "SampledRequestsEnabled": true,
        "CloudWatchMetricsEnabled": true,
        "MetricName": "AWS-AWSManagedRulesBotControlRuleSet"
      }
    }
]
EOF

Update the AWS WAF web ACL with the rule:

aws wafv2 update-web-acl \
  --name WAF-FOR-YELB \
  --scope REGIONAL \
  --id $WAF_WAF_ID \
  --default-action Allow={} \
  --lock-token $(aws wafv2 list-web-acls \
    --region $WAF_AWS_REGION \
    --scope REGIONAL \
    --query "WebACLs[?Name=='WAF-for-Yelb'].LockToken" \
    --output text) \
  --visibility-config SampledRequestsEnabled=true,CloudWatchMetricsEnabled=true,MetricName=YelbWAFAclMetrics \
  --region $WAF_AWS_REGION \
  --rules file://waf-rules.json

After waiting about 10 seconds, test the rule by sending a request:

curl $(kubectl get ingress yelb.app -n yelb \
  -o jsonpath="{.status.loadBalancer.ingress[].hostname}")

testing the rule via request
AWS WAF should block the request, and requests sent using curl should now receive a 403:

receiving 403 from curl request

If you request the page using a web browser, AWS WAF will allow the request. Let’s verify it. Get the ALB’s DNS name:

echo $(kubectl get ingress yelb.app -n yelb \
  -o jsonpath="{.status.loadBalancer.ingress[].hostname}")

Open the URL in your browser and you should see Yelb UI.

the Yelb UI dashboard

Note that we added AWSManagedRulesBotControlRuleSet rule group to AWS WAF web ACL (see configuration file waf-rules.json). This rule group contains rules to block and manage requests from bots as described in AWS WAF documentation. AWS WAF blocks the requests we send using curl because AWS WAF web ACL rules are configured to inspect and block requests for user agent strings that don’t seem to be from a web browser.

AWS WAF logging and monitoring

Network security teams require AWS WAF logging to meet their compliance and auditing needs. AWS WAF provides near-real-time logs through Amazon Kinesis Data Firehose. AWS WAF logs each request along with information such as timestamp, header details, and the action of the rule that matched. Customers can integrate AWS WAF logs with Security information and event management (SIEM) solutions or other log analysis tools for debugging and forensics. You can enable access logging in AWS WAF, save AWS WAF logs to Amazon S3, and use Amazon Athena to query WAF logs without creating servers. AWS WAF also allows you to redact certain fields during logging, which is helpful if your requests contain sensitive information that should not be logged.

After implementing an AWS WAF, it is critical to regularly review your applications’ traffic to develop a baseline understanding of its traffic patterns. Application and security teams should review AWS WAF metrics and dimensions to ensure that the web ACL rules block requests that can potentially compromise the application’s security and availability.

AWS Shield Advanced and WAF

AWS Shield Advanced subscribers can also engage the AWS Shield response team during an active DDoS attack. The AWS Shield Response team helps you analyze suspicious activity and assists you in mitigating the issue. The mitigation often involves updating or creating AWS WAF rules and AWS WAF web ACLs in your account.

AWS Firewall Manager

AWS Firewall Manager enables customers that operate multiple AWS accounts to centrally manage their web ACL. It simplifies administration and maintenance tasks across multiple accounts and resources for a variety of protections, including AWS WAF, AWS Shield Advanced, Amazon VPC security groups, AWS Network Firewall, and Amazon Route 53 Resolver DNS Firewall.

If you’d like to use AWS Firewall Manager to centralize the control of AWS WAF in multiple AWS accounts, you’d also need:

  1. AWS Organizations: Your organization must be using AWS Organizations to manage your accounts, and All Features must be enabled. For more information, see Creating an organization and Enabling all features in your organization.
  2. A Firewall Manager administrator account: You must designate one of the AWS accounts in your organization as the Firewall Manager administrator for Firewall Manager. This gives the account permission to deploy security policies across the organization.
  3. AWS Config: You must enable AWS Config for all of the accounts in your organization so that Firewall Manager can detect newly created resources. To enable AWS Config for all of the accounts in your organization, use the Enable AWS Config template from the StackSets sample templates.

You can associate Firewall Manager with either a management account or a member account that has appropriate permissions as a delegated administrator. AWS Organizations’ documentation includes more information about using Firewall Manager with AWS Organizations. 

Cleanup

Use the following commands to delete resources created during this post:

kubectl delete ingress yelb.app -n yelb
aws wafv2 disassociate-web-acl \
  --resource-arn $(aws wafv2 list-resources-for-web-acl \
                     --web-acl-arn $WAF_WACL_ARN \
                     --region $WAF_AWS_REGION \
                     --query 'ResourceArns' --output text) \
  --region $WAF_AWS_REGION
aws wafv2 delete-web-acl --id $WAF_WAF_ID --name  WAF-FOR-YELB --scope REGIONAL \
    --lock-token $(aws wafv2 list-web-acls \
      --region $WAF_AWS_REGION \
      --scope REGIONAL \
      --query "WebACLs[?Name=='WAF-for-Yelb'].LockToken" \
      --output text) \
    --region $WAF_AWS_REGION
helm delete aws-load-balancer-controller -n kube-system   
eksctl delete iamserviceaccount --cluster $WAF_EKS_CLUSTER_NAME --region $WAF_AWS_REGION --name aws-load-balancer-controller
aws iam detach-role-policy --policy-arn $WAF_LBC_IAM_POLICY_ARN --role-name $(aws iam list-entities-for-policy  --policy-arn $WAF_LBC_IAM_POLICY_ARN --query 'PolicyRoles[0].RoleName' --output text)
aws iam delete-policy --policy-arn $(echo $(aws iam list-policies --query 'Policies[?PolicyName==`AWSLoadBalancerControllerIAMPolicy-WAFDEMO`].Arn' --output text))
eksctl delete cluster --name $WAF_EKS_CLUSTER_NAME --region $WAF_AWS_REGION

Conclusion

This post demonstrates how to protect your web workloads using AWS WAF. Amazon EKS customers benefit from AWS WAF-provided AWS Managed Rules to add a web application firewall to web apps without learning how to write AWS WAF rules. Additionally, AWS WAF Bot Control gives you visibility and control over common and pervasive bot traffic that can consume excess resources, skew metrics, cause downtime, or perform other undesired activities.

We recommend implementing a AWS WAF and testing its effectiveness by conducting penetration tests regularly to identify gaps in your AWS WAF rules. The Guidelines for Implementing AWS WAF whitepaper provides a detailed implementation guide for anyone looking to protect web applications.

Elamaran Shanmugam

Elamaran Shanmugam

Elamaran (Ela) Shanmugam is a Sr. Container Specialist Solutions Architect with AWS. Ela is a Container, Observability and Multi-Account Architecture SME and helps customers design and build scalable, secure and optimized container workloads on AWS. His passion is building and automating infrastructure to allow customers to focus more on their business. He is based out of Tampa, Florida and you can reach him on twitter @IamElaShan.

Re Alvarez-Parmar

Re Alvarez-Parmar

In his role as Containers Specialist Solutions Architect at Amazon Web Services, Re advises engineering teams with modernizing and building distributed services in the cloud. Prior to joining AWS, he spent more than 15 years as Enterprise and Software Architect. He is based out of Seattle. Connect on LinkedIn at: linkedin.com/in/realvarez/