Fireworks.ai Delivers 4x Throughput for Generative AI and Cuts Latency by up to 50% Using AWS and NVIDIA

The CEO and cofounder of Fireworks.ai, Lin Qiao, had led the development and productionization of the open-source, popular deep learning framework PyTorch while she served as senior director of engineering at Meta. In fact, many in the Fireworks.ai founding team worked on PyTorch to empower developers in the building of fast, secure, affordable generative AI products. “Achieving optimal cost-performance for scale and productionization is a primary challenge for customers developing on PyTorch,” says Dmytro Dzhulgakov, Fireworks.ai cofounder and chief technology officer. “It’s particularly true with generative AI products and models because of their sheer size as well as how new and fast this field is. We wanted to use AWS to help to bridge this gap.”

Fireworks.ai’s inference solution both hosts generative AI software-as-a-service and supports containerized deployments in a customer’s virtual private cloud. Developers can choose to use state-of-the-art, open-source language, image, and multimodal foundation models off the shelf, or they can customize and fine-tune models. For example, Fireworks.ai supports Stable Diffusion XL, a deep learning, text-to-image model; Llama 2 large language models with up to 70 billion parameters; and StarCoder, designed specifically for code-related tasks.

From its inception, Fireworks.ai has collaborated closely with AWS teams to determine how best to optimize compute power for its highly demanding inference engine. At first, the company used Amazon EC2 P4d Instances, powered by NVIDIA A100 Tensor Core GPUs, which offer high performance for machine learning training and high-performance computing applications in the cloud. In July 2023, AWS announced Amazon EC2 P5 Instances, the next generation of Amazon EC2 P4 instances and the highest-performance GPU-based instances for deep learning and high performance computing applications. “AWS has the latest and greatest hardware,” Dzhulgakov says.

Using Amazon EC2 P5 Instances, Fireworks.ai meets demanding customer requirements for performance. For example, one customer lowered the latency of its summarization model—which uses natural language processing techniques to condense written content—by 30–50 percent. “We support about three times higher traffic with a single instance,” says Dzhulgakov. “Even though the instance itself is more expensive if you calculate the cost per single request, there actually ends up being overall cost improvement.” In fact, the customer cut total costs by four times.

Another Fireworks.ai customer, Sourcegraph, uses StarCoder, which runs on Fireworks.ai, to power inference and output for the open-source edition of its AI coding assistant, Cody. After Fireworks.ai adopted Amazon EC2 P5 Instances, Cody doubled its completion acceptance rate, a measure of a model’s output efficiency. Additionally, Cody’s backend latency accelerated by more than two times. “Using AWS, Fireworks.ai helps developers integrate powerful open models into their prototype applications without breaking the bank as they experiment, explore, and play with different models,” Dzhulgakov says. “As their product grows and usage increases, the focus shifts toward speed and cost. Using Amazon EC2 P5 Instances, we provide outstanding cost per performance for our customers’ use cases.”

And because the Fireworks.ai inference solution built on AWS is HIPAA and SOC2 Type II compliant, it meets important considerations for companies with strict security requirements for their data. “Using AWS as an infrastructure provider does help a lot in achieving certification for our solution,” says Dzhulgakov.