Charting the life of an Amazon CloudFront request

Amazon CloudFront is a native AWS Content Delivery Network (CDN) service. CDNs provide web acceleration by using a worldwide network of edge locations closer to end-users, and caching content at the edge. However, CloudFront can do a lot more than that, with functionality at the edge to do geo-filtering, execute functions, perform AWS Web Application Firewall (WAF) filtering, among others. In this post, we explore the life of a client request to a CloudFront distribution, paying particular attention to the order of execution of these features. This understanding is essential in optimizing your web application delivery, securing web applications, and troubleshooting your CDN configuration.

Before we begin the journey, we can explore the infrastructure pieces involved in a CloudFront client request.

Figure 1: CloudFront Edge locations and Regional Edge Caches

Edge caching overview

CloudFront points of presence (POPs), also called Edge locations, are the first group of servers that a request hits, and they are responsible for either responding to a request (if cached) or forwarding it to the next layer. They are globally distributed across the world and have a smaller footprint than a traditional AWS Region. You can consider a POP as a single unit for logical purposes, clarity, and organization. Figure 1 (referenced from the official CloudFront documentation) demonstrates this point.

This streamlining works well, but sometimes we may need to dive deeper into the request-response flow to troubleshoot CDN configuration issues, optimize caching, or improve dynamic content delivery performance. We notice that the viewer request and response path follow different layers within the CloudFront network. The POPs are responsible for the initial connection handling, request load-balancing, caching, and CloudFront Functions execution, while the Regional Edge Caches (REC)s are responsible for more cache optimization, execution of Lambda@Edge, edge-to-origin connections, request collapsing, and origin timeout configuration. You can enable an optional Origin Shield feature to improve cache efficiency.

Along with the HTTP(s) protocol, CloudFront also supports enhancements to the protocol such as gRPC, an open source remote procedure call (RPC) framework built on top of HTTP/2, and Web Sockets, a TCP-based protocol that is useful for long-lived bidirectional connections between clients and servers where a persistent connection is a requirement for real-time applications.

We only focus on the HTTP(s) request and response handling for this post and explore gRPC and Web Sockets connections in later posts.

DNS resolution and POPs

Our journey begins with a user visiting a website behind CloudFront, as shown in the following figure. Sites are commonly setup using a custom hostname to a specific CloudFront domain name. From the DNS request, CloudFront determines the location of the user, and provides a DNS response with the optimal edge location to handle this request. CloudFront uses several metrics such as internet networks health, network load, and other factors to provide viewers with multiple IP addresses of the optimal POP. Depending on the locations of your end users, you have the option to limit the infrastructure that responds to requests, thereby saving money and using different Price Classes. The price class chosen in the CloudFront distribution limits the list of POPs available to the user. Users can use the capabilities of CloudWatch Network Monitor and CloudWatch Internet Monitor to gain operational visibility into the network and internet performance and availability of your applications hosted on AWS.

The process that uses DNS resolution to determine the ideal CloudFront POP is different if the user is using CloudFront with Anycast static IPs. This post assumes non-anycast IPs.

Figure 2: Path of a CloudFront request

Connection establishments and TLS negotiation

When the DNS resolution is complete, the client application, a web browser or a mobile application (referred to as viewer), receives a set of IP addresses of the optimal POPs. The client application initiates a connection to a POP using one of those IPs, and it can handle failover using any of the other IPs if needed. CloudFront accepts HTTP, HTTPs, and WebSockets, on port 80/443, in accordance with IETF standards. Each POP has AWS Shield Standard protection against common DDoS volumetric attacks (UDP floods, SYN floods). The next layer makes sure that the Secure Sockets Layer (SSL)/Transport Layer Security (TLS) connection is established correctly. The security policy configured for the CloudFront distribution indicates the protocols and ciphers that are accepted.

Request routing and validation

The request is handed off to a request router. The request router at the POP load-balances client connections across multiple cache servers. There is also a critical security layer that monitors and protects the cache servers by making sure that the request from the client is Requests for Comments (RFC) compliant and does not pose any threat through illegal/ambiguous syntax. This layer makes sure that the requests forwarded to the caching layers are properly formatted and HTTP specification compliant. At this point, the accepted protocol, verb and geo-restrictions are evaluated based on the CloudFront distribution configuration.

AWS WAF

Following the request load-balancing and pre-access security, if AWS WAF is enabled for the CloudFront distribution, then the request is processed through the rules enabled in the AWS WAF Web Access Control List (WebACL). AWS WAF is a web application firewall that monitors the requests to your application to protect them from application layer attacks such as SQL injection, cross-site scripting, bot attacks, or DDoS attacks. AWS WAF always executes before any content-handling rules such as cache-behaviors, request/response header policies, or Edge Compute functions such as CloudFront Functions or Lambda@Edge.

Behaviors

At this point, users can define the way that CloudFront handles the request, under the Behaviors section. Behaviors can be different for different path patterns. Behaviors specify the origin to use, the allowed HTTP methods, cache policy, function associations, and origin request policy, which determines which parameters (headers, query strings, and cookies) are forwarded to the origin. Users can also configure field-level encryption to protect sensitive information.

Edge functions

On the viewer side (before cache), users can choose between two types of functions at the edge: CloudFront Functions or Lambda@Edge. A CloudFront Function is a serverless function that executes on the POP. CloudFront Functions are lightweight functions in JavaScript for high-scale, latency-sensitive CDN customizations. Lambda@Edge is an extension of AWS Lambda, and it can execute functions in Node.js or Python at the REC. If the CloudFront distribution configuration has a viewer-side Lambda@Edge code, then the request is forwarded to the REC. Lambda@Edge always executes at the RECs. The same is true for PUT, POST, and DELETE HTTP methods, because they are dynamic requests. Viewer Request functions can modify all existing origin capabilities, such as the primary origin, custom headers, and timeouts. The execution takes place before the first cache layer, which means they are executed for every request.

CloudFront caching

CloudFront queries the cache at the POP after the CloudFront Functions viewer request (if any). Within a POP there are several caching layers to maximize the cache-hit ratio. If the first layer does not have the object in cache, then the request is forwarded to the next layer, and the next. We cannot have an infinite number of layers, thus there is a limit to the number of cache servers in each cache stack or peers to which the first layer reaches out. If the final layer of cache within a POP does not have the object in cache, then the request is forwarded to the REC.

REC

The REC has similar caching layers to improve cache footprint, capacity, and the necessary computing infrastructure to execute the Lambda@Edge functions. RECs act as a secondary macro-level caching layer between the POP and the origin to further improve cache-hit ratio, reduce requests to the origin, and as infrastructure for Lambda@Edge code execution.

If you defined a Lambda@Edge Viewer Request function, then it is executed at this point, before CloudFront looks at the cache in the REC. After that, if the object is not found in the REC cache, then any Lambda@Edge origin request is executed.

If the CloudFront distribution has Origin Shield enabled, all RECs connect to an Origin Shield before sending a request to the origin, thus reducing the load on the origin. The Origin Shield is located closer to the user origin and improves caching efficiency by reducing traffic bandwidth and request volume to the origin.

The final layer on the origin-connection side (REC or Origin Shield) maintains persistent connections with the content origin to achieve high data-transfer efficiency. Origin timeout settings (for custom origins) enable users to adjust the following values:

1. Connection attempts: You can set the number of times that CloudFront attempts to connect to the origin.
2. Connection timeout: Number of seconds that CloudFront waits when trying to establish a connection to the origin.
3. Response timeout: How long CloudFront waits for a response after forwarding a request to the origin and CloudFront waits after receiving a packet of a response from the origin and before receiving the next packet.

Origin request policy also determines the minimum SSL version to be used by the Edge when connecting to the origin.

Origin response

If the request is not found in any of the caching layers, the REC, or the Origin Shield, then it is retrieved from the origin. The origin is a resource reachable over a public IP, but it can also be a private resource if used in conjunction with VPC origins. If the origin is a URL, then DNS resolution takes place at this point. That allows you to use Amazon Route 53 routing policies, such as latency-based routing or geolocation routing, to determine the location to use.

The response follows the request path in reverse. The response from the origin is sent back to the REC. If the request is cacheable and compression is enabled, then the response is compressed. Caching duration is managed through the CloudFront behavior cache policy. If the Lambda@Edge origin response code is defined, then it is executed at this point, and cached at the REC. If a Lambda@Edge viewer response function is defined, then it is executed. For security reasons, the functions executed over the response do not have access to read the response body, but they can replace it. The journey continues at the POP. If a CloudFront Functions viewer response function is defined, then it is executed at the POP, and the final asset is delivered to the client. Figure 2 summarizes the main steps in the request/response journey.

Conclusion

In this post we followed the journey of a single request from a viewer to Amazon CloudFront, to the origin server, and the response from the origin server back to the viewer, thus exploring the different layers and functionality that CloudFront provides.

Now that you have a better understanding of the order of execution, and where each feature/functionality resides, we encourage you to review your CloudFront configuration for cache settings, edge functions, AWS WAF and AWS Shield, and use all the power of CloudFront CDN.

About the authors