Palo Alto Networks Boosts 2,000 Developers’ Productivity Using AI Solutions from AWS, Anthropic, and Sourcegraph

Based in California, Palo Alto Networks is a multinational company that provides organizations with best-in-class cybersecurity solutions. When generative AI entered the market, Palo Alto Networks knew the technology had the potential to transform its business operations—but the question was how to use it in a way that would benefit internal teams as well as customers. Recognizing that the quality of its products—and customer satisfaction—begins with the first line of code, leadership determined that integrating a generative AI solution into its development teams’ flow of work would be the best use of this opportunity. The goal was to create a generative AI tool that would bring velocity and quality to its 3,500 developers worldwide. Ideally, a generative AI tool would allow them to deliver more features in the same timeframe, optimize code to be more performant and secure, and help developers troubleshoot faster.

However, the security of its source code, the company’s crown jewel, was of the utmost importance, and the security team’s data privacy requirements eliminated most products. Additionally, because generative AI solutions were so new, they were rapidly changing or didn’t always deliver on promises—and few offered comprehensive integrations. Palo Alto Networks understood it could better control the security, customization, and iteration if its engineers designed a custom solution. But building a solution from the ground up, in-house, didn’t make sense. “A company whose whole mission is to build and solve these issues for customers is always going to solve this optimally. By not creating everything from the ground up, we could remain focused on building great security products,” said Gunjan Patel, director of engineering at Palo Alto Networks.

Knowing that they wanted to create a generative AI solution using state-of-the-art language models and services, the first step for a core team of engineers was to assess dozens of options on the market against an internal evaluation rubric. The plan was to develop a proof of concept and roll it out to globally distributed teams strategically. This would require close collaboration with its selected vendors. Finally, the team had to determine how it would measure the success of its endeavor, a challenging task given the subjectivity of what makes better code.

Palo Alto Networks ranked AI coding assistants and large language models (LLMs) against benchmarks, including code architecture, code generation, unit test generation, code documentation, and debugging or error detection. It also weighed pros and cons around security, scalability, performance, and customizations. Palo Alto Networks then selected three providers that met their security and scaling criteria and ranked the highest in the evaluations: AWS, Sourcegraph, and Anthropic. For the solution’s foundation, Palo Alto Networks chose Amazon Bedrock, a fully managed service offering high-performing foundation models. At the core of the solution, the team integrated Anthropic’s Claude 3.5 Sonnet and Claude 3 Haiku LLMs, hosted on Bedrock, with Sourcegraph's Cody, an AI coding assistant. Cody is a tool integrated into developers' IDEs (Integrated Development Environments), where they write, debug, and maintain code. It connects with Claude models configured within Palo Alto Networks' Bedrock environment, creating a seamless and unified system for developers.

Palo Alto Networks chose AWS because the cloud provider was a trusted third party that could host Anthropic and Sourcegraph, providing the cybersecurity company with a secure, single-tenant infrastructure. Because Sourcegraph’s deployment options allow for flexibility in hosting infrastructure and LLM endpoint providers, Palo Alto Networks chose to deploy it in a single-tenant, self-hosted configuration on AWS. Cody, the AI coding assistant, was configured to invoke Amazon Bedrock private VPC endpoints provided by Palo Alto Networks. This allowed Palo Alto Networks the security and throughput to run Anthropic’s Claude models for its top-secret data. “All three companies did a lot of customization to meet our requirements, and everything came together,” Patel said.

With shared excitement and a willingness to collaborate closely, the engineering teams architected a solution and proof of concept within three months. The team designed the solution so that Cody, hosted in Palo Alto Networks' AWS account, invokes Amazon Bedrock, where the Claude models are hosted, creating a seamless integration between Cody and the Claude models for enhanced code development and maintenance. This design enabled Palo Alto Networks to build and scale its generative AI coding assistant application. Developers interact with Cody on the front end via a web user interface or IDE extensions. When a developer feeds Cody a prompt through a chat interface, Cody appends additional context and sends the request to Claude. Claude then processes the query and generates a response, which Cody displays back to the user.

Palo Alto Networks first tested the prototype for a month before rollout, starting with low-stakes, open-source code projects. From there, leadership selected 150 core product developers working in different roles, countries, and programming languages to continue piloting the tool. After two months of iterative testing and feedback, Palo Alto Networks opened it to more than 1,000 developers. To maximize effectiveness, developers receive targeted training from AWS, Anthropic, and Sourcegraph. Anthropic delivers prompt engineering workshops, Sourcegraph runs Cody learning sessions, and AWS holds AI trainings. “AWS, Anthropic, and Sourcegraph worked with us throughout the entire project, participating in everything from joint testing to operations support,” Patel said.

The new solution serves as a central place to ask questions and generate secure, quality code for numerous use cases—across different programming languages and working with thousands of APIs. For example, the engineering team built a tool that uses the existing Prisma Cloud static application security testing (SAST) capabilities to detect and use Cody for automatic remediation of code vulnerabilities. Just three months after Palo Alto Networks completed the initial architecture, the company onboarded 2,000 developers to its generative AI coding tool, with the goal of adding 1,500 more in the following quarter. In that time, leadership saw coding productivity increase up to 40 percent, with the company-wide average between 20 and 25 percent. This is in part because the centralized chat interface reduces the time developers spend searching through documentation for answers, among other things.

Thanks to custom commands that support broad use cases, increasing internal adoption, and further sophistication of the solution, Palo Alto Networks’ leadership is aiming to double the number of features the company can deliver in the same amount of time. “We asked a new hire to write an integration, which required them to understand the product, APIs, code base, and write the code. With this solution, a new developer ramped up on an unfamiliar codebase quickly—and started contributing to it on the first day,” Patel said.

As Palo Alto Networks is looking forward to realizing greater long-term benefits—such as how this initiative translates to fewer defects for customers—the company is expecting productivity and code quality to grow over time as the tool gets smarter and more use cases materialize. “We are taking a land-and-expand approach, working closely with Palo Alto Networks' product teams to jointly develop specific use cases tailored to the customer's needs. From this initial engagement, we are co-developing a new use case with the customer's Internet of Things (IoT) team as a starting point for our expanded collaboration,” said Sean Blakley, principal AWS sales at AWS.

Amazon Bedrock is a fully managed service that offers a choice of high-performing foundation models (FMs) from leading AI companies like AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon through a single API, along with a broad set of capabilities you need to build generative AI applications with security, privacy, and responsible AI.