The Inference server offers the full infrastructure to run fast inference on GPUs.

It includes llama.cpp inference, latest CUDA and NVIDIA Docker container toolkit.

Leverage the multitude of models freely available to run inference with 8 bit or lower quantized models which makes inference possible on e.g. 16 GB or 24 GB memory GPUs.

Llama.cpp offer efficient inference of quantized models in interactive and server mode. It features

• Plain C/C++ implementation without dependencies
• 2-bit, 3-bit, 4-bit, 5-bit, 6-bit and 8-bit integer quantization support
• Running inference on GPU and CPU simultaneously allowing to run larger models in case GPU memory is insufficient
• AVX, AVX2 and AVX512 support for x86 architectures
• Supported models: LLaMA, LLaMA 2, Falcon, Alpaca, GPT4All, Chinese LLaMA / Alpaca and Chinese LLaMA-2 / Alpaca-2, Vigogne (French), Vicuna, Koala, OpenBuddy (Multilingual), Pygmalion 7B / Metharme 7B, WizardLM, Baichuan-7B and its derivations (such as baichuan-7b-sft), Aquila-7B / AquilaChat-7B, Starcoder models, Mistral AI v0.1, Refact

• llama-cpp-python: OpenAI API compatible Llama.cpp inference server
• Open Interpreter: let language models run code on your computer. An open-source, locally running implementation of OpenAIs Code Interpreter.
• Tabby coding assistant: a self-hosted AI coding assistant, offering an open-source alternative to GitHub Copilot