Introducing Amazon CodeWhisperer Dashboard and CloudWatch Metrics

Amazon CodeWhisperer is an AI coding companion that helps improve developer productivity by offering contextualized code snippets, or suggestions, generated based on coding context and natural language prompts to help developers build software quickly, responsibly, and securely. For enterprise customers, CodeWhisperer offers a Professional tier that adds administrative features, like SSO and IAM Identity Center integration, policy control for code suggestions with references to open-source code, and higher limits on security scanning.

When I work with individual developers, they immediately recognize the productivity gains CodeWhisperer offers. However, it can be difficult to measure that same impact across the entire organization. Enterprise customers have requested to understand how many of their developers use CodeWhisperer, how often they use it, and how often they accept the suggestions that CodeWhisperer offers. This is now possible via the CodeWhisperer Dashboard and metics that CodeWhisperer sends to Amazon CloudWatch enabling enterprises to measure the impact of CodeWhisperer.

In this post, I will walk through the new metrics and dashboard options. Note that these features are only available in CodeWhisperer Professional. If you have not configured CodeWhisperer Professional, you can follow the instructions in the CodeWhisperer User Guide. For the remainder of this post, I will assume that you have configured CodeWhisperer Professional are logged into the AWS Console as a CodeWhisperer Administrator. Note that when using the CodeWhisperer Professional, CodeWhisperer does not collect your content for service improvement purposes.

Using the CodeWhisperer Dashboard

CodeWhisperer now includes a dashboard, shown in the following image, to help enterprises understand how developers are benefiting from CodeWhisperer. This dashboard is designed to answer the most common questions.

The CodeWhisperer Dashboard includes the following sections:

User activity: The User activity section includes the active subscriptions and daily active users. Active subscriptions indicate how many CodeWhisperer seats you have provisioned, while daily active users indicate how many of those seats are used on a daily basis.

Code impact: The Lines of code generated by CodeWhisperer indicates how many lines of code were suggested by CodeWhisperer and accepted by your developers. The percentage of code written by CodeWhisperer indicates how much of your total code is made up of suggestions from CodeWhisperer. This is a great indication of the value your developers are gaining from the service. Accepted suggestions with references indicates the number of suggestions from CodeWhisperer that developers accepted which are based on open-source projects, the references to which CodeWhisperer makes available to you.

Security scans: CodeWhisperer security scan is a tool that helps identify security vulnerabilities in your developers’ code. The data shown indicates how many scans your developers have run in their Integrated Development Environment (IDE).

Using CloudWatch Metrics

While the CodeWhisperer Dashboard will address the most common questions, CodeWhisperer records additional data that may be of interest. CodeWhisperer records these as Amazon CloudWatch metrics, allowing you to create alarms, and custom dashboards. CodeWhisperer records CloudWatch metrics with various levels of granularity.

First, CodeWhisperer records a few high-level metrics as shown in the following image. Most of these already appear on the Dashboard, including the SubscriptionCount, ActiveUsers and Invocations. However, it is still useful to have them in CloudWatch so you can set alarms and build your own dashboards. In the following image, I show the weekly invocation count over the past two months.

Diving a little deeper, you may want to understand how CodeWhisperer is used for specific languages. For example, I may want to understand how CodeWhisperer benefits the front-end developers, who use JavaScript, and compare that with the back-end developer, who use Python. I can do this by leveraging the Programming Language dimension as shown in the following image.

Finally, if you want to dive even deeper, you could can use the more granular CompletionType, ProgrammingLanguage, SuggestionState dimension. CompletionType can either be BLOCK or LINE. Depending on your use case, CodeWhisperer may generate an entire function block in one suggestion or provide line-by-line suggestion. SuggestionState can be ACCEPT indicating that the user accepted the suggestion, REJECT indicating the user rejected the suggestion, or DISCARD indicating that CodeWhisperer did not make the suggestion.

I could use these metric on their own, but it is often even more powerful to use CloudWatch Math Metric to calculate custom metrics. For example, in the following image, I calculate the accept rate for JavaScript suggestions by dividing the sum of accepted suggestions by the total number of accepted and rejected suggestions.

CodeWhisperer Customizations

Swami Sivasubramanian recently announced Amazon CodeWhisperer customization capability to generate more relevant code suggestions informed by your organization’s code base. This capability uses the latest model and context customization techniques and is now available in preview. With this capability, CodeWhisperer has added additional CloudWatch metrics that let you focus on your customizations. In addition, you can filter the CodeWhisperer dashboard by customization, as shown in the following image.

Of course, you can also filter by customization in CloudWatch. CodeWhisperer publishes additional metrics that include the CustomizationARN dimension. You can use these new metrics to gather details on how your customizations are performing. For example, in the following image, I am graphing the accepted suggestions for Java for my CodeWhisperer customization. I can use this to understand how well the custom version of CodeWhisperer is performing and look for opportunities to improve my customizations.

If you did not get the desired performance improvement, consider increasing the data size and data quality for the customization.

Custom Dashboards

While CloudWatch metrics are interesting on their own, you may want to create a custom dashboard that displays multiple metrics on the screen at once. For example, I would like to see the accept rate and percent of code Written by CodeWhisperer for each of the languages my team uses. These include Python, Java, JavaScript and TypeScript. In the following, image have combined each of these on a single dashboard so I track it once place.

You can download the source for this dashboard in JSON format. If you would like to create your own dashboard, you can learn more by reading, Creating a CloudWatch Dashboard in the CloudWatch documentation. Alternatively, you can see an example of creating a custom CodeWhisperer Dashboard using CDK in this GitHub repository.

Conclusion

In the past, it was difficult to measure CodeWhisperer’s impact across the entire organization. CodeWhisperer now support CloudWatch Metrics along with a new CodeWhisperer Dashboard that provides insightful data about how your developers use the service. These new features provide the tools to understand how your organization is benefitting from CodeWhisperer. You can read more in the CodeWhisperer Documentation.

About the Author