A/B testing

A/B testing or canary deployments technique allow developers to experiment with two or more variants of a web page. Variants are randomly shown to users, and then statistical analysis is used to determine which variant performs better for a given business goal. For example, a UI variant of a product page can be tested with the goal of increasing product sales.

You can implement A/B testing in different ways according to your technical and business requirements. One of the main technical decisions is where to execute the variant selection logic: on client side, on edge server side (at CDN level) or on origin server side.

Origin server-side variant selection. In this approach, A/B selection is executed directly on the origin server hosting your web application. While this approach is CDN agnostic, it's incompatible with CDN caching or with static file hosting. It's an option for Server-Side Rendered web applications that are fully dynamic.

Edge server-side variant selection. In this approach, you make the variant selection decision on the CDN level. This approach requires minimal to no changes on the application, and is compatible with CDN caching. In CloudFront, you can implement the A/B testing logic using edge functions. An Edge function can use the request attributes (country, cookie, etc..), in addition to external A/B testing API to select one of the variants and have CloudFront serve it from the cache.

Client-side variant selection. In this approach, your frontend first makes a request to an A/B testing API to decide which variant to serve, then based on the response, download the variant, then render it in the browser. In this approach, your A/B testing is CDN agnostic, however it's tightly coupled to your application and introduces additional latency due to additional steps on the client-side. Note also that it's not always an option, such as for A/B testing statically generated sites (SSG). To implement client-side A/B testing, you can use CloudWatch Evidently. It provides the client SDK and the UI to control A/B testing experiments. For hands-on tutorial on CloudWatch Evidently, consider this workshop.

Let's focus on Edge server-side A/B testing. To design the best implementation for your business, consider the additional questions:

• Do your require stickiness (e.g. the same user will always get the same variant)? Stickiness is usually implemented using cookies.
• What dimensions are used to select a variant for a user? country, user-id, etc..
• How frequently do you do A/B testing? Heavy usage of A/B testing with many experiments running in parallel by different teams, requires a more sophisticated implementation compared to simpler, occasional A/B testing.

To learn about some of the edge service-side implementations of A/B testing using CloudFront's edge functions, consider this hands-on this workshop. In the following section, we will share two common implementations of A/B testing using CloudFront, from the aforementioned workshop.

For occasional A/B testing needs, such as quarterly design improvement of an institutional website, consider a CloudFront Functions based solution for the duration of your experiment. The solution is based on two functions configured on the Cache behavior that matches the page for which you want to run A/B testing:

• A viewer request function, that inspects the experiment cookie value of incoming requests, and based on it rewrites the URL to the selected page variant. If the cookie is not present, the function selects the variant using your custom logic, such as 60% for variant A and 40% for variant B, only for requests coming from France.
• A viewer response function, that sets the cookie on the client based the selected variant, if the cookie was not already present. The experiment cookie is used to make sure a single user always receives the same variant of the page to avoid breaking their web experience.

Changing the variant selection logic, such as to increase the percentage of Variant B, requires an update of the viewer event's function code, which generally takes seconds to complete. When you are done with the A/B testing experiment, you can remove the edge functions to save costs.

If you have a continuous practice of A/B testing on your website, with experiments conducted on a daily basis, you can still use the previous CloudFront Function based solution. However, it's recommended to decouple the variant selection parameters (e.g. variant weights) from the function code. You can use a CloudFront KeyValueStore to store such parameters, outside of the function code. The KeyValueStore is ideal as a global (every PoP), low-latency read,
key value datastore at scale (millions of requests per second).

Note that your use case should respect the KeyValueStore quotas, such as the maximum size (5 MB) or change velocity limit of few API calls per second.

Consider a Lambda@Edge based solution if a CloudFront Function based solution does not meet your A/B testing need, for example if the KeyValueStore quotas are limiting for your use case. This can be the case if you are operating a large e-commerce website, with many teams executing simultaneous experiments on different parts of the website at a high velocity.  In this scenario, use Lambda@Edge instead of CloudFront Function, with an external data source (e.g. DynamoDB Global Tables) for storing the A/B testing parameters. Feature Management tools like LaunchDarkly provide integrations with DynamoDB to persist the A/B testing parameters.

• Workshop: Client-side A/B testing using Evidently
• Workshop: Edge server-side A/B testing workshop using CloudFront's edge functions
• TrueCar Case study: Edge server-side A/B testing using Lambda@Edge, DynamoDB and LaunchDarkly
• Ryanair Case study: Edge server-side A/B testing using Lambda@Edge, DynamoDB and LaunchDarkly