Kubernetes Gateway API in action

As modern applications grow in complexity, connectivity in Kubernetes environments demands sophisticated routing scenarios that often exceed the capabilities of traditional Kubernetes objects. Historically, the Kubernetes framework has relied on services for internal traffic and Ingress resources for external traffic management. Although these primitives allow for basic traffic flow scenarios, more advanced use cases such as canary testing and egress controls often necessitate the adoption of third-party solutions. This results in adopting Custom Resource Definitions (CRD) and annotating existing resources with vendor specific configurations. The latter is especially true for Service Mesh implementations that lack a standardized API to route traffic within and to external targets. Furthermore, different personas such as Cluster Operators or Developers need to operate on single Kubernetes objects, such as Ingress resources, which causes issues while enforcing proper boundary of responsibilities.

In this post, we explore advanced traffic routing patterns with the Kubernetes Gateway API through an example, demonstrating how it streamlines and standardizes application connectivity and service mesh integration in Kubernetes.

Unified traffic handling use cases

In this post, we use a Calendar web application, which we name Cal-LLM, which allows us to create a schedule for conferences. The initial version of the application only allowed users to enter their sessions manually. However, developers have been working on a new generative AI powered feature that allows users to automatically fill in session information and summarize abstracts with a third-party large language model (LLM), powered by Mistral AI, as shown in the following figure.

The setup consists of a presentation layer (web-app), which communicates with two versions of the calendar microservice (calendar and calendar-v2), where the new version (calendar-v2) connects to Mistral AI LLM. For this architecture, we use Linkerd as our Service Mesh solution. Both the presentation layer and calendar microservice have the Linkerd sidecar injected. Furthermore, we make use of Envoy Gateway as the controller to work with Gateway API resources.

We recommend using the ExternalDNS resource to streamline the process of keeping your domain names in sync with your Kubernetes resources and automatically manage DNS records in your specified DNS provider (in this case, Amazon Route 53).

This architecture enables us to explore three key use cases of Kubernetes Gateway API:

1. Exposing applications: In our example, we use an AWS Network Load Balancer (NLB) routing external traffic to the presentation layer of the application (web-app). This represents the classic north-south traffic pattern, where the Gateway API can manage things such as external traffic distribution, handle TLS termination, and implement rate limiting and authentication policies. In this post we provide an example of handling request routing based on hostnames.
2. Canary deployments with gRPC: A notable use case is the canary deployment pattern between the web service and two calendar versions (calendar and calendar-v2). The support from the Gateway API for gRPC traffic makes it particularly valuable to manage things such as progressive rollouts of new calendar versions, protocol-aware routing for gRPC services, and health checking and automatic failover. In this post we showcase traffic splitting between service versions.
3. Egress traffic control: The architecture demonstrates egress traffic flow from calendar-v2 to Mistral AI LLM. In general, for egress traffic, the Gateway API can control and monitor outbound traffic, implement security policies for external service access, provide service mesh capabilities for east-west communication, and enable traffic metrics and observability.

These use cases demonstrate the versatility of the Gateway API when handling different traffic patterns within a Kubernetes environment, from traditional HTTP/HTTPS traffic to modern gRPC communications and controlled external service access. The unified approach streamlines configuration and maintains consistency across different traffic types and patterns.

Expose applications to external clients (Ingress)

The Kubernetes Gateway API introduces a powerful and elegant way to manage external access to your applications, offering sophisticated routing capabilities through a declarative, Kubernetes-native approach. In this section we explore a practical example of hostname-based routing for North-South traffic handling.

The core of this functionality is demonstrated in the HTTPRoute configuration:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: web-app
spec:
  parentRefs:
  - name: eg    # References the Gateway, spec below
  hostnames:
  - "www.cal-llm.com"
  rules:
  - matches:
    - hostname: www.cal-llm.com
    backendRefs:
    - name: web-app
      port: 8080

The API seamlessly integrates with the underlying infrastructure through the Gateway resource, which can be configured to work with existing AWS load balancers and target groups.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"

When using Amazon Elastic Kubernetes Service (Amazon EKS), the Gateway resource can be annotated with AWS Load Balancer Controller specific annotations that control the provisioning of Load Balancer resources. In our example, we bind our Envoy Proxy, which has been provisioned through the Gateway object and processed by the Envoy Gateway controller, to an internet-facing NLB. Incoming traffic is routed to targets within the cluster based on the rules defined inside of the HTTPRoute resources attached to the Gateway resource. In this specific example, incoming traffic with the host header www.cal-llm.com is routed from the Envoy Proxy to the web-app service. Given the separation of the routing layer abstracted with the HTTPRoute resource and the infrastructure layer implemented by the Gateway resource, developer and cluster operators can work on different layers of the application connectivity stack without interfering with each other.

Reliable connection between microservices (East-West)

Along with the handling of North-South use cases, the Kubernetes Gateway API can be used for connectivity and traffic management between services inside the cluster (East-West traffic). This allows reusability of the same resources (for example GRPCRoute and HTTPRoute) for this use case. A Service Mesh is needed to implement the scenario—we use Linkerd.

In this example we enable connectivity between the presentation layer pod (web-app) and the backend versions of the calendar applications (calendar and calendar-v2). Although the new calendar-v2 version has gone through rigorous internal testing, it is best practice to introduce it to a subset of users first and gradually increase exposure to a bigger audience in a canary deployment fashion. The Kubernetes Gateway API provides advanced traffic routing functionalities that allow for common deployment patterns such as A/B or canary. Rather than referencing an ingress proxy such as in the previous example, we need to bind the target resource to a Kubernetes Service object. In our example, we reference the calendar Service exposed on port 8080. Furthermore, we specify multiple backends (calendar and calendar-v2) to which we want to route traffic based on a given percentage. For our scenario we are using a GRPCRoute because the presentation layer of our application (web-app) communicates to the backend service over GRPC.

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: GRPCRoute
metadata:
  name: calendar
  namespace: apps
spec:
  parentRefs:
    - name: calendar
      kind: Service
      group: core
      port: 8080
  rules:
    - backendRefs:
        - name: calendar
          port: 8080
          weight: 40
        - name: calendar-v2
          port: 8080
          weight: 60

When we deploy this resource, it gets parsed and translated by the Linkerd control plane. Subsequently, proxies that are part of our Service Mesh get automatically configured with the rules outlined in the GRPCRoute. Incoming traffic that hits the calendar Service is split and routed among calendar and calendar-v2 with a 40 to 60 distribution.

Usage of the Kubernetes Gateway API inside the cluster is enabled by the Gateway API for Service Mesh (GAMMA) working group.

Restrict communication to external services (Egress)

Without putting more controls in place, Pods running inside of the cluster can communicate with external endpoints without restrictions, which can lead to security issues such as data exfiltration. Gateway API provides native support to control egress traffic from applications that are part of the Service Mesh. These rules are enforced on the proxies themselves and can be bypassed with the right permissions. Therefore, we recommend adding more protections such as native Network Policies to strengthen the Security Perimeter.

To restrict egress traffic for the individual workload, we use the HTTPRoute custom resource once again. Similar to the examples in the North-South and East-West scenarios, we need to reference the HTTPRoute with a suitable parentRef. For the egress use case, this needs to be a resource of the type EgressNetwork. An egress network can be specified cluster-wide or on a namespace level, and it defines the default traffic policy for the Pods that are in scope.

apiVersion: policy.linkerd.io/v1alpha1
kind: EgressNetwork
metadata:
  namespace: apps
  name: deny-all
spec:
  trafficPolicy: Deny

When we deploy this resource into the apps’ Namespace, the Linkerd proxies are configured to block outgoing connections. To allow access from our calendar service to the externally hosted Mistral AI API, we need to create policies to allow access to the API from our workload. We can achieve that by using familiar Gateway API resources such as HTTPRoute, TLSRoute, or GRPCRoute. The API exposing our LLM uses HTTPS, thus we use the TLSRoute. Readers should choose the right resource based on the scheme of their application’s outgoing traffic.

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TLSRoute
metadata:
  name: allow-mistral-ai-access
  namespace: apps
spec:
  hostnames:
  - 
  parentRefs:
  - name: deny-all
    kind: EgressNetwork
    group: policy.linkerd.io
    namespace: apps
    port: 443
  rules:
    - matches:
      - path:
          value: "/"

This TLSRoute that we defined allows egress traffic to the api.mistral.ai domain and further restricts access on the /v1/chat/completions path. This resource is bound to the previously created EgressNetwork policy that denied the traffic. Now pods in the mesh that are within the apps namespace can connect to the Mistral AI endpoint.

This implementation is Linkerd specific, and other Service Meshes might have other Egress controls implementations or lack this functionality completely. We suggest consulting the official documentation for the given vendor for more information.

Summary

In this post we demonstrated through a practical example how the Kubernetes Gateway API enables users to unify traffic management with three scenarios: host-based North-South, canary-based East-West, and host-based egress filtering. We also underlined how this new set of APIs adds a lot on top of the Ingress’ capabilities. For example, cluster administrators and developers can use a single set of common APIs such as HTTPRoute, GRPCRoute, or TLSRoute for the majority of their traffic patterns. Moreover, in most cases, they can avoid using vendor specific annotations to enable advanced functionalities such as weight-based routing, circuit breaking, or retries.

We encourage you to try out this new specification to enable advanced application networking use cases in your Kubernetes environment. A good place for beginning familiarizing yourself with more advanced concepts is the official documentation.

About the authors

Valentin Widmer is a Senior Cloud Architect at Amazon Web Services. He works with customers providing them technical guidance and helping them to build secure, scalable and highly available solutions using AWS Cloud services. His interests include Application Modernisation, Data Engineering and Generative AI.

Federica Ciuffo is a Senior Specialist Solutions Architect for Containers at Amazon Web Services. She is specialized in container services and is passionate about building infrastructure with code. Outside of the office, she enjoys reading, drawing, and spending time with her friends, preferably in restaurants trying out new dishes from different cuisines.