Composio Increases AI Coding Agent Accuracy by 50% with Multi-Model Testing on Amazon Bedrock

Founded in India in 2023, Composio provides an AI integration platform that connects LLMs with external tools and services. The platform helps developers and enterprises develop AI agents that integrate seamlessly across different software environments, making AI-powered automation more efficient.

Initially, Composio managed separate integrations for LLMs across different providers, requiring developers to switch between multiple SDKs and APIs. This fragmented approach added complexity, making experimentation less efficient, while the lack of a centralized testing framework further restricted iteration speed. Meanwhile, an enterprise customer required a more efficient way to evaluate multiple AI models for automation workflows. “We and our clients needed a way to experiment rapidly with multiple AI models without the overhead of managing separate integrations,” recalls Soham Ganatra, co-founder and CEO at Composio.

Recognizing the need for a more efficient, unified approach to AI model evaluation, the company sought a way to streamline experimentation and support multiple LLMs without adding operational complexity.

After evaluating various AI model deployment and testing solutions, Composio chose AWS for its scalability, security, and ability to centralize AI model testing. By adopting Amazon Bedrock, a generative AI solution, Composio empowered its developers to test multiple AI models in parallel and optimize performance more efficiently. “With Amazon Bedrock, we can experiment faster and optimize our AI models with minimal operational overhead,” explains Ganatra. The company also utilized Amazon Bedrock to determine that Anthropic Claude delivered the highest accuracy for its coding agent.

Beyond providing a scalable testing platform, AWS helped facilitate smooth deployment at scale—while ensuring security and compliance requirements were met. AWS’s SOC Type II compliance gave Composio confidence in handling data securely as it optimized AI-driven workflows. Initially, low throughput limits caused request throttling that prevented running large-scale experiments. AWS engineers worked closely with the Composio team to increase token throughput from five million to ten million per minute, enabling seamless parallel testing. “AWS worked closely with us to resolve throughput limitations, which allowed us to scale AI model testing without disruptions,” Ganatra adds.

Shifting AI model experimentation to Amazon Bedrock eliminated operational bottlenecks and cut testing time by two weeks, helping Composio’s engineering team iterate and optimize solutions faster. With model testing and comparison centralized, workflows were streamlined and AI agent performance scaled more effectively.

 “The unified testing platform gave us a huge efficiency boost and helped us make data-driven decisions faster,” says Ganatra. As a result, Composio’s AI-powered coding agent—built with SWE-Kit—increased accuracy by 50 percent, automated engineering tasks, reduced errors, and enhanced overall efficiency.

These advancements led to Composio securing the top ranking on SWE-Bench, a leading benchmark for evaluating AI-assisted code generation and problem-solving. This recognition validated the model’s accuracy and effectiveness compared to other AI-powered coding tools, strengthening Composio’s credibility. The platform’s success also attracted over 450 new developers and secured a major enterprise customer seeking AI-powered automation.

Additionally, support for parallelized experimentation has given Composio a competitive edge. “With AWS, we can scale experiments quickly to deliver high-performance AI automation solutions,” Ganatra concludes. “Amazon Bedrock enhances our model selection, reduces latency, and drives continuous performance improvements.”