Best Practices for the Custom Lens Lifecycle: Measure and Improve

In this blog post, we present best practices and resources to help you build, validate, and improve an AWS Well-Architected Custom Lens. The AWS Well-Architected Custom Lens is a feature of the AWS Well-Architected Tool that lets you bring your own content and best practices complementing the existing Well-Architected Framework that allow you to assess the quality of your AWS workloads. Because of its versatility, you can use the Custom Lens to (1) bring the best practices of the AWS Well-Architected Lenses and AWS Guides and Whitepapers to the Well-Architected Tool, or (2) create your own content in the tool to test your organization’s unique architectures and workloads.

In a previous blog post, we presented the Custom Lens Lifecycle, a process to create and continuously improve a Custom Lens for your organization. The lifecycle consists of four phases: (1) plan, (2) implement, (3) measure, and (4) improve which you repeat to achieve the optimal custom content quality. The quality improvement promotes a willingness to perform more frequent reviews, you will receive more data to develop a more comprehensive view of your workload. Ultimately, this improves the overall adherence to best practices and requirements in your organization.

Based on the insights from collaborations with customers, we have compiled a list of best practices that you can apply when implementing the Custom Lens lifecycle in your company. In a previous blog post, we presented best practices for the plan and implement phases. Below, we will present best practices for the measure phase.

Measure Phase: Best Practices

Perform a Trial as an Early Feedback Loop

Once you create the first version of the Custom Lens, work with a small group of test users or teams to test the Lens. Then survey the team to gather their feedback on its performance. This will act as a checkpoint to reflect on the efficacy of the created content, before more effort is spent. Ideally, testers have already worked with the Well-Architected Framework, in order to identify how the Custom Lens complements the existing content provided by AWS. Here is how you can conduct such a trial:

1. Select a small number of teams for the first round. It is enough to have around five teams, as you will already get sufficient feedback for refinement.
2. If you have not done so already, move your best practices to the specified Custom Lens JSON format.
3. Prepare a survey that participants have to fill out after they worked with the Custom Lens. We will provide a list of example questions at the end of this section.
4. Upload the JSON in the Well-Architected Tool in a designated management account. This AWS account is typically owned by the platform team or CCoE.
5. Invite the teams for an onboarding session. Explain that as a prerequisite, teams need to have access to the Management Console of their AWS account.
6. During the session, explain the goals of the Custom Lens. Then, ask the teams to share their account IDs so that the owner of the management account can share the Custom Lens with their AWS accounts.
7. After the Custom Lens has been shared, ask teams to create a new workload in the Well-Architected Tool and select the Custom Lens.
8. Ask the teams to assess the selected workload using the Custom Lens for a specified time period. Afterwards, ask them to fill out the survey on their impressions.

After each team has responded to the survey, evaluate the results and use the feedback to determine the next steps. You might come to the conclusion that the Custom Lens is already achieving the target outcomes, or that you need some refinement of the content. Here is an example survey we created with one of our customers:

Please indicate with a number from 1 to 5 (best) how much you agree with each statement unless other options are given.

5: Strongly agree
4: Somewhat agree
3: Neither agree or disagree
2: Somewhat disagree
1: Strongly disagree

Collaborate with Executive Stakeholders, Platform and Developer Teams for continuous improevment

A Custom Lens should be created in the open: In large organizations, a lot of know-how and best practices can sit isolated in certain lines of business or product teams, as so-called tribal knowledge. Working on the Custom Lens in an open forum can be an effective mechanism to surface this know-how and scale it out for the whole organization.

It can also encourage other teams to adopt the Custom Lens. Once you have successfully performed your first trial and you are satisfied with the survey results, you should plan an introductory session for your whole organization. In that session, you should explain the value of your Custom Lens: It is an easily accessible source for company-specific best practices and patterns for developing software.

Make it clear that your Custom Lens is open for comments and improvements, and explain how to make contributions – for example, by sending a pull request to the Git repository where the JSON file is stored. Use this feedback to improve the Custom Lens and release updated versions regularly.

Summary and Next Steps

In this blog post, we presented best practices for the measure phase of the Custom Lens lifecycle and how to improve a custom lens over time. Do you have a special software development lifecycle, a customized security and compliance framework, or other highly specific requirements or best practices that you want to make visible to teams and measurable throughout your company? Then check out the original feature announcement blog post to learn more about how to create a Custom Lens in the Well-Architected Tool, check out the overview of the Custom Lens lifecycle, read the best practices for the plan and implement phases, and apply the best practices presented here throughout your organization.