Sold by: Hanwei Software Technology
Deployed on AWS
This product has charges associated with it for integration of the NVIDIA GPU environment, vLLM 0.28.0 runtime, preloaded Qwen2.5 3B model, OpenAI-compatible API configuration, and tested Ubuntu 24.04 deployment on g4dn.xlarge with Tesla T4.
Overview
vLLM on Ubuntu 24.04 LTS provides a production-ready high-performance LLM inference environment. Built on Ubuntu 24.04.4 LTS with Long Term Support until 2034, this AMI includes vLLM inference server and pre-installed Qwen2.5 7B model optimized for GPU inference.
vLLM is developed by the Sky Computing Lab at UC Berkeley and uses PagedAttention technology to achieve state-of-the-art throughput for LLM serving. It supports continuous batching, tensor parallelism, and provides an OpenAI-compatible API.
This AMI is designed for enterprises and developers who need to deploy AI models at scale without relying on third-party cloud services. All inference happens locally on your AWS GPU instance, ensuring data privacy and low latency.
Key components included:
- Ubuntu 24.04.4 LTS (Long Term Support until 2034)
- NVIDIA Driver 580.x (Tesla T4, V100, A100 support)
- CUDA Toolkit 12.5 (GPU computation framework)
- vLLM 0.8.x (LLM inference server, Apache 2.0 licensed)
- PyTorch 2.x (Deep learning framework, Apache 2.0 licensed)
- Qwen2.5 7B Instruct (Pre-installed, Apache 2.0 licensed)
The system supports multiple deployment scenarios:
- OpenAI-compatible API: Serve models via REST API on port 8000
- Streaming responses: Support for streaming token generation
- Tensor parallelism: Scale to larger models across multiple GPUs
Recommended use cases:
- Production LLM serving at scale
- Enterprise AI API endpoints
- Privacy-sensitive inference workloads
- Cost-effective AI inference (pay for instance time only)
Highlights
- High-performance inference - PagedAttention technology delivers 2-4x throughput compared to traditional LLM serving. Continuous batching optimizes GPU utilization.
- Production-ready - Pre-installed Qwen2.5 7B model. Start serving immediately with OpenAI-compatible API on port 8000.
- Privacy-first - All AI inference runs locally on your instance. No data leaves your infrastructure. Ideal for sensitive enterprise workloads.
Details
Sold by
Delivery method
Delivery option
64-bit (x86) Amazon Machine Image (AMI)
Latest version
Operating system
Ubuntu 24.04 LTS
Deployed on AWS
New
Introducing multi-product solutions
You can now purchase comprehensive solutions tailored to use cases and industries.
Features and programs
Financing for AWS Marketplace purchases
AWS Marketplace now accepts line of credit payments through the PNC Vendor Finance program. This program is available to select AWS customers in the US, excluding NV, NC, ND, TN, & VT.
Pricing
Pricing is based on actual usage, with charges varying according to how much you consume. Subscriptions have no end date and may be canceled any time. Alternatively, you can pay upfront for a contract, which typically covers your anticipated usage for the contract duration. Any usage beyond contract will incur additional usage-based costs.
Additional AWS infrastructure costs may apply. Use the AWS Pricing Calculator to estimate your infrastructure costs.
Dimension | Cost/hour |
|---|---|
g4dn.xlarge Recommended | $350.00 |
g4dn.2xlarge | $350.00 |
g4dn.4xlarge | $350.00 |
g4dn.8xlarge | $350.00 |
g4dn.12xlarge | $350.00 |
g4dn.16xlarge | $350.00 |
g4dn.metal | $350.00 |
p3.2xlarge | $350.00 |
p3.8xlarge | $350.00 |
p3.16xlarge | $350.00 |
Vendor refund policy
No Refund
Custom pricing options
Request a private offer to receive a custom quote.
Legal
Vendor terms and conditions
Upon subscribing to this product, you must acknowledge and agree to the terms and conditions outlined in the vendor's End User License Agreement (EULA) .
Content disclaimer
Vendors are responsible for their product descriptions and other product content. AWS does not warrant that vendors' product descriptions or other product content are accurate, complete, reliable, current, or error-free.
Delivery details
64-bit (x86) Amazon Machine Image (AMI)
Amazon Machine Image (AMI)
An AMI is a virtual image that provides the information required to launch an instance. Amazon EC2 (Elastic Compute Cloud) instances are virtual servers on which you can run your applications and workloads, offering varying combinations of CPU, memory, storage, and networking resources. You can launch as many instances from as many different AMIs as you need.
Version release notes
High-performance LLM inference server AMI built on Ubuntu 24.04 LTS with vLLM and pre-installed Qwen2.5 7B model. Powered by PagedAttention technology for efficient GPU memory management.
Included Software:
- Ubuntu 24.04.4 LTS (Base OS)
- NVIDIA Driver 580.x (GPU driver)
- CUDA Toolkit 12.5 (GPU computation)
- vLLM 0.8.x (LLM inference server, Apache 2.0 licensed)
- PyTorch 2.x (Deep learning framework, Apache 2.0 licensed)
- Qwen2.5 3B Instruct (Pre-installed model, Apache 2.0 licensed)
Features:
- High-throughput inference with PagedAttention
- OpenAI-compatible API (v1 /chat/completions)
- Continuous batching for optimal throughput
- Tensor parallelism support for larger models
- Pre-installed model ready to serve
Security:
- SSH key pair authentication only
- Automatic security updates enabled
- Run vLLM as non-root user
Optimizations:
- Pre-installed Qwen2.5 7B model ready to serve
- CUDA environment variables pre-configured
- Optimized memory allocation for GPU inference
Additional details
Usage instructions
SSH to the instance and login as 'ubuntu' using the key pair specified at launch. For more details on connecting to a Linux instance, see: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstancesLinux.html
Once connected, view the built-in README for complete instructions:
- Run: cat ~/ami-readme.md
The README includes:
- How to start the vLLM server
- How to use the OpenAI-compatible API
- How to serve additional models
- GPU verification commands
Quick start:
- Check GPU: nvidia-smi
- Start server: vllm serve qwen/qwen2.5-7b-instruct
- API endpoint: http://localhost:8000/v1/chat/completions
For more details, refer to the README.
Support
Vendor support
If you encounter problems in the process of using the system, please feel free to contact us by email: support@thinkclouds.ai . Thank you!
AWS infrastructure support
AWS Support is a one-on-one, fast-response support channel that is staffed 24x7x365 with experienced and technical support engineers. The service helps customers of all sizes and technical abilities to successfully utilize the products and features provided by Amazon Web Services.
Similar products
This is a repackaged open source software product wherein additional charges apply for a hardened, AWS-optimized image with verification reports and maintenance. This AMI delivers vLLM 0.24 as a bring-your-own-model OpenAI-compatible inference server, on an Ubuntu 24.04 LTS base tuned for EC2 by Easycloud-with a provenance evidence pack and SBOM.
This is a repackaged software product wherein additional charges apply for a pre-hardened, SI Core STIG Hardened image and seller support. Ubuntu 24.04 LTS is designed for enterprise environments, providing a robust and secure platform for the deployment of cloud applications. With its long-term support and regular updates, Ubuntu 24.04 LTS ensures stability and reliability for critical workloads. Seamlessly integrate Ubuntu 24.04 LTS into your existing infrastructure, benefiting from its extensive package repository and developer-friendly tools. Ideal for container orchestration, web services, and high-performance computing, Ubuntu 24.04 LTS equips organizations with the flexibility to innovate while maintaining a strong security posture. Choose Ubuntu 24.04 LTS for modern cloud solutions.
This product has charges associated with it for CoreNova hardening, maintenance, validation notes, and seller support. Ubuntu 24.04 LTS Hardened for AWS Graviton (ARM64, LVM, XFS) provides a hardened Ubuntu 24.04 LTS EC2 baseline with SSH lockdown, audit logging, AIDE, firewall controls, and buyer-side OpenSCAP notes.
This product has charges associated with it for Hanwei's integration of TensorFlow 2.21, PyTorch 2.5.1, CUDA 12.6, cuDNN 9.x, NVIDIA driver 595, Python 3.12, Jupyter, and scientific libraries on Ubuntu 24.04 LTS, plus GPU configuration and testing for NVIDIA T4 on g4dn.
Customer reviews
No customer reviews yet
Be the first to review this product . We've partnered with PeerSpot to gather customer feedback. You can share your experience by writing or recording a review, or scheduling a call with a PeerSpot analyst.