We use essential cookies and similar tools that are necessary to provide our site and services. We use performance cookies to collect anonymous statistics, so we can understand how customers use our site and make improvements. Essential cookies cannot be deactivated, but you can choose “Customize” or “Decline” to decline performance cookies.
If you agree, AWS and approved third parties will also use cookies to provide useful site features, remember your preferences, and display relevant content, including relevant advertising. To accept or decline all non-essential cookies, choose “Accept” or “Decline.” To make more detailed choices, choose “Customize.”
Customize cookie preferences
We use cookies and similar tools (collectively, "cookies") for the following purposes.
Essential
Essential cookies are necessary to provide our site and services and cannot be deactivated. They are usually set in response to your actions on the site, such as setting your privacy preferences, signing in, or filling in forms.
Performance
Performance cookies provide anonymous statistics about how customers navigate our site so we can improve site experience and performance. Approved third parties may perform analytics on our behalf, but they cannot use the data for their own purposes.
Allowed
Functional
Functional cookies help us provide useful site features, remember your preferences, and display relevant content. Approved third parties may set these cookies to provide certain site features. If you do not allow these cookies, then some or all of these services may not function properly.
Allowed
Advertising
Advertising cookies may be set through our site by us or our advertising partners and help us deliver relevant marketing content. If you do not allow these cookies, you will experience less relevant advertising.
Allowed
Blocking some types of cookies may impact your experience of our sites. You may review and change your choices at any time by selecting Cookie preferences in the footer of this site. We and selected third-parties use cookies or similar technologies as specified in the AWS Cookie Notice.
Your privacy choices
We and our advertising partners (“we”) may use information we collect from or about you to show you ads on other websites and online services. Under certain laws, this activity is referred to as “cross-context behavioral advertising” or “targeted advertising.”
To opt out of our use of cookies or similar technologies to engage in these activities, select “Opt out of cross-context behavioral ads” and “Save preferences” below. If you clear your browser cookies or visit this site from a different device or browser, you will need to make your selection again. For more information about cookies and how we use them, read our Cookie Notice.
To opt out of the use of other identifiers, such as contact information, for these activities, fill out the form here.
For more information about how AWS handles your information, read the AWS Privacy Notice.
Unable to save cookie preferences
We will only store essential cookies at this time, because we were unable to save your cookie preferences.
If you want to change your cookie preferences, try again later using the link in the AWS console footer, or contact support if the problem persists.
A/B testing or canary deployments technique allow developers to experiment with two or more variants of a web page. Variants are randomly shown to users, and then statistical analysis is used to determine which variant performs better for a given business goal. For example, a UI variant of a product page can be tested with the goal of increasing product sales.
Architectural decisions
You can implement A/B testing in different ways according to your technical and business requirements. One of the main technical decisions is where to execute the variant selection logic: on client side, on edge server side (at CDN level) or on origin server side.
Origin server-side variant selection. In this approach, A/B selection is executed directly on the origin server hosting your web application. While this approach is CDN agnostic, it's incompatible with CDN caching or with static file hosting. It's an option for Server-Side Rendered web applications that are fully dynamic.
Edge server-side variant selection. In this approach, you make the variant selection decision on the CDN level. This approach requires minimal to no changes on the application, and is compatible with CDN caching. In CloudFront, you can implement the A/B testing logic using edge functions. An Edge function can use the request attributes (country, cookie, etc..), in addition to external A/B testing API to select one of the variants and have CloudFront serve it from the cache.
Client-side variant selection. In this approach, your frontend first makes a request to an A/B testing API to decide which variant to serve, then based on the response, download the variant, then render it in the browser. In this approach, your A/B testing is CDN agnostic, however it's tightly coupled to your application and introduces additional latency due to additional steps on the client-side. Note also that it's not always an option, such as for A/B testing statically generated sites (SSG). AWS AppConfig enables you to dynamically deploy your application configuration changes to your clients.
Let's focus on Edge server-side A/B testing. To design the best implementation for your business, consider the additional questions:
Do your require stickiness (e.g. the same user will always get the same variant)? Stickiness is usually implemented using cookies.
What dimensions are used to select a variant for a user? country, user-id, etc..
How frequently do you do A/B testing? Heavy usage of A/B testing with many experiments running in parallel by different teams, requires a more sophisticated implementation compared to simpler, occasional A/B testing.
To learn about some of the edge service-side implementations of A/B testing using CloudFront's edge functions, consider this hands-on this workshop. In the following section, we will share two common implementations of A/B testing using CloudFront.
Common use cases using Edge server-side A/B testing
Occasional A/B testing
For occasional A/B testing needs, such as quarterly design improvement of an institutional website, consider a CloudFront Functions based solution for the duration of your experiment. The solution is based on two functions configured on the Cache behavior that matches the page for which you want to run A/B testing:
A viewer request function, that inspects the experiment cookie value of incoming requests, and based on it rewrites the URL to the selected page variant. If the cookie is not present, the function selects the variant using your custom logic, such as 60% for variant A and 40% for variant B, only for requests coming from France.
A viewer response function, that sets the cookie on the client based the selected variant, if the cookie was not already present. The experiment cookie is used to make sure a single user always receives the same variant of the page to avoid breaking their web experience.
Changing the variant selection logic, such as to increase the percentage of Variant B, requires an update of the viewer event's function code, which generally takes seconds to complete. When you are done with the A/B testing experiment, you can remove the edge functions to save costs.
Image not found
Frequent A/B testing
If you have a continuous practice of A/B testing on your website, with experiments conducted on a daily basis, you can still use the previous CloudFront Function based solution. However, it's recommended to decouple the variant selection parameters (e.g. variant weights) from the function code. You can use a CloudFront KeyValueStore to store such parameters, outside of the function code. The KeyValueStore is ideal as a global (every PoP), low-latency read, key value datastore at scale (millions of requests per second).
Note that your use case should respect the KeyValueStore quotas, such as the maximum size (5 MB) or change velocity limit of few API calls per second.
CapitalOne case study: A/B testing using CloudFront Functions and KeyValueStore
Advanced A/B testing
Consider a Lambda@Edge based solution if a CloudFront Function based solution does not meet your A/B testing need, for example if the KeyValueStore quotas are limiting for your use case. This can be the case if you are operating a large e-commerce website, with many teams executing simultaneous experiments on different parts of the website at a high velocity. In this scenario, use Lambda@Edge instead of CloudFront Function, with an external data source (e.g. DynamoDB Global Tables) for storing the A/B testing parameters. Feature Management tools like LaunchDarkly provide integrations with DynamoDB to persist the A/B testing parameters.
