Artificial Intelligence
Author: Omri Shiv
Omri Shiv is a Senior Open Source Machine Learning Engineer focused on helping customers adopt and best utilize open source tools for their AI/ML journey. He maintains the AI on EKS project and the Agentic Community.
Serving LLMs using vLLM and Amazon EC2 instances with AWS AI chips
The use of large language models (LLMs) and generative AI has exploded over the last year. With the release of powerful publicly available foundation models, tools for training, fine tuning and hosting your own LLM have also become democratized. Using vLLM on AWS Trainium and Inferentia makes it possible to host LLMs for high performance […]