Cutting AI Service Costs Using Amazon Bedrock with rabbit

Released in April 2024, rabbit r1 is a handheld mobile device that is controlled by voice, image, and text input. Its operating system, called rabbit OS, can recognize complex user intentions and automate actions such as scheduling events, summarizing meetings, and translating between languages. The system depends on its large action model (LAM)—a cross-solution general AI agent system.

Previously, the company was using a different cloud provider to run its AI language and vision assistant workloads. But after the debut of r1 and with the first deliveries of preorders approaching, rabbit needed more reliable and scalable infrastructure. The company needed to support an influx of new users on its LAM with a seamless out-of-the-box experience and solid security. “We were prepared to support 3,000 users, but we sold 100,000 units of r1,” says Nathaniel Waisbrot, head of the platform team at rabbit. “We needed close partnership with our cloud provider to meet those scaling challenges. With a tight window, we couldn’t afford friction in onboarding or implementing the necessary changes.”

Working alongside cloud services company Caylent, rabbit aimed to enhance latency, increase rate quotas, expand geographic support, and improve customer service while optimizing costs. Caylent has cloud and AI expertise, skilled engineers, and partnership with AWS, all of which helped rabbit plan a large-scale migration to AWS. The teams orchestrated a fast migration so that rabbit could scale its LAM seamlessly and maintain high performance as more than 100,000 devices arrived in users’ hands. The companies migrated rabbit’s infrastructure to Amazon EKS in 2 months, and rabbit met its deadline for the product launch.

By migrating to Amazon EKS, rabbit has significantly improved its system performance and reliability. To enhance the experience for r1 users, the company also optimized its AI stack using Amazon Bedrock, a fully managed service for building and scaling generative AI applications with foundation models. To upgrade the AI-powered capabilities of rabbit r1, rabbit adopted Anthropic’s Claude in Amazon Bedrock, a service for building generative AI solutions with Anthropic’s state-of-the-art model, Claude. rabbit is using AWS to optimize the speed and cost-effectiveness of its product’s AI functionalities, including application interfacing. And by using Amazon Bedrock, rabbit was able to effectively reduce its AI service costs.

“Scaling rabbit has been a remarkable engineering effort and also one of the most rewarding,” says Randall Hunt, CTO at Caylent. “The infrastructure had to go from zero to millions of interactions in weeks, and our automatic scaling features achieved that. When rabbit launched in New York, our whole team was monitoring Kubernetes logs and seeing the real-time traffic spike. It was incredible.”

The company also accelerated the voice-operation response times so that users can have a seamless experience interacting with r1. Caylent helped rabbit dramatically reduce its text-to-speech latency on AWS so that r1 users can enjoy a reliable and responsive experience. “Combining the comprehensive set of AWS services with Caylent’s matching know-how, we can effortlessly use outstanding solutions across our stack, from large language model APIs and text-to-speech functionality to low-latency API security systems,” says Waisbrot.

In addition, Caylent helped rabbit capitalize on opportunities to automate its infrastructure management, increasing resilience and scalability and accelerating deployment. To reduce manual infrastructure code writing and management, rabbit adopted Terraform, an infrastructure-as-code tool for creating, updating, and versioning AWS infrastructure. Thus, rabbit’s engineers could focus on rapid AI and software development to deliver ongoing updates for users.

The companies implemented advanced security measures, including service control policies, and compliance protocols on Amazon EKS. They calibrated rabbit’s account-provisioning framework to prevent unauthorized access and block malicious IP addresses using AWS WAF, a service for protecting web applications from common bugs and bots. On AWS, rabbit is delivering a seamless, low-latency experience for users of its innovative AI-powered device, providing them with cost efficiency and advanced security features.

rabbit is continuing to refine its LAM and expand the r1 feature set. It developed a system to securely deliver over-the-air updates to its r1 devices using AWS Internet of Things, a service for unlocking Internet of Things data and accelerating business growth. Going forward, rabbit will focus on fleet management and continue prioritizing global scalability with robust security. To support its expanding customer base, rabbit is also working with Caylent to implement a multicluster, cell-based architecture on AWS. And by using Amazon Bedrock, rabbit has been able to rapidly experiment with alternate models for automation—such as Anthropic’s beta feature for Claude, Computer Use.

“AWS and Caylent provided us with nimble and trusted support. Their high level of commitment is important for startup success, particularly in an emerging field such as AI,” says Zhuoheng Yang, founding engineer at rabbit.