Overview
Mango LLMBoost is a containerized solution empowering LLM experts with tools to optimize their models and dynamically select the most suitable GPUs for their workloads. By fine-tuning and optimizing the LLM inference engine, Mango LLMBoost fully harnesses the parallelism of GPU cores and orchestrates inference jobs to maximize utilization across all available GPUs. Through quantization that utilizes the smaller data format without loss in accuracy, Mango LLMBoost further enhances efficiency by ensuring effective use of high-speed GPU caches and memory.
Highlights
- High-performance & Cost-efficient: Speed up your inference by up to 12x and cut costs by up to 92% over other inference frameworks.
- Flexible: Mango LLMBoost works on all popular NVIDIA and AMD GPUs.
- Hassle-free: End-to-end deployment options with our Web-serving and streaming APIs.
Details
Typical total price
$31.131/hour
Features and programs
Financing for AWS Marketplace purchases
Pricing
Free trial
Instance type | Product cost/hour | EC2 cost/hour | Total/hour |
|---|---|---|---|
p4d.24xlarge | $1.00 | $32.773 | $33.773 |
p5.48xlarge | $1.00 | $98.32 | $99.32 |
g6e.48xlarge Recommended | $1.00 | $30.131 | $31.131 |
Additional AWS infrastructure costs
Type | Cost |
|---|---|
EBS General Purpose SSD (gp3) volumes | $0.08/per GB/month of provisioned storage |
Vendor refund policy
No refunds are issued.
Legal
Vendor terms and conditions
Content disclaimer
Delivery details
64-bit (x86) Amazon Machine Image (AMI)
Amazon Machine Image (AMI)
An AMI is a virtual image that provides the information required to launch an instance. Amazon EC2 (Elastic Compute Cloud) instances are virtual servers on which you can run your applications and workloads, offering varying combinations of CPU, memory, storage, and networking resources. You can launch as many instances from as many different AMIs as you need.
Version release notes
Please contact us at contact@manogboost.io for release notes.
Additional details
Usage instructions
Launching
Use the web interface to launch the AMI on one of the supported instances. Once the instance is running, you can connect to it using an SSH client with the private key used to launch the instance. For example:
Ssh -i --privatekey.pem-- ubuntu@--instance-ip-or-dns--The default username of the instance is ubuntu.
Getting Started
This AMI contains the MangoBoost LLMBoost Application and SDK.
The tutorial applications can be run with:
python tutorial.pySupport
Vendor support
AWS infrastructure support
AWS Support is a one-on-one, fast-response support channel that is staffed 24x7x365 with experienced and technical support engineers. The service helps customers of all sizes and technical abilities to successfully utilize the products and features provided by Amazon Web Services.
Similar products
