LiteLLM LLM Gateway - Self Hosted (requires Private Offer)

Our main use case for LLM Gateway  is that our company has partnerships with multiple LLM providers including OpenAI, Claude, and Gemini . LLM Gateway  acts as an interface between all three providers. I would describe it as a router that functions as middleware between our application and the AI providers so that we do not need to give or share API keys to each team.

Our team calls LLM Gateway from their application, and all the keys and routing configurations are present in LLM Gateway. Its responsibility is to connect with Claude, OpenAI, or Gemini  based on the request we receive.

We have an application in which users can ask anything. For example, if a user is asking a general question, we call LLM Gateway and pass the model name as ChatGPT. It internally uses ChatGPT itself. If the question is related to the application we created, it internally uses RAG and goes to Claude. LLM Gateway is responsible for redirecting the request based on context.

LLM Gateway also has an additional feature where if one of the models is unavailable at a time, it automatically redirects the request to another model, so there is no downtime in the application. The automatic failover feature ensures that if one model is not available, LLM Gateway redirects the request to another model.

The best features LLM Gateway offers include multi-provider AI access and the ability to access around 200 plus models available in the market. We just need to pass our key and set up this one, and it can access all the available models. Apart from this, it automatically routes the request based on context if we set it in LLM Gateway. Another feature is the automatic failover functionality where if something goes wrong, it redirects the request to another model. LLM Gateway also provides usage analytics with a dashboard where we can check the current usage of each model and see how many requests are going to each model. It persists data for around 30 days, so we can review usage over the last month. LLM Gateway can be self-hosted as well, which is beneficial for large companies with security concerns.

I find multi-provider access and failover to be the most valuable features day-to-day. Multi-provider access integrates all available models, acting as a router between the application and LLM Gateway. If my application is using four different models, I only need to call LLM Gateway, which manages everything. We also do not need to share sensitive API keys, as the developer can directly call LLM Gateway, which handles everything seamlessly. The failover feature automatically redirects requests if something goes wrong in one model, and it is incredibly easy to configure. It does not take more than a minute to set up.

One positive impact of LLM Gateway on my organization is reducing security risk. If we give API keys to everyone, they can misuse them outside the organization. However, we no longer share API keys, as users just need to call our LLM Gateway, and the API keys remain secret and contained within our on-premises setup. Security-wise, it has significantly reduced our organization's risk.

Regarding improvements, I think the pricing can be more competitive. LLM Gateway takes 5% of the token usage, which feels a bit high. While they do have a free tier, the costs for the enterprise edition are somewhat high. As a new product in the market, it should charge less compared to competitors. However, I think the cost is comparable or slightly higher.

I have been using LLM Gateway for around 1.5 years. It is a new product in the market.

LLM Gateway is stable. It is a new product, but it is heading in the right direction.

Currently, we are using around 20 models, and it works fine. LLM Gateway claims integration with around 200 models, but we have only utilized 20 in our organization so far.

The customer support and documentation for LLM Gateway are pretty good. Although the community is a bit sparse because of its newness, the available support is very effective. I rate the customer support a perfect 10.

I previously used Portkey  in a different organization but have not switched from Portkey  to LLM Gateway within the same organization.

I usually start with the free tier, which is very good. For the enterprise version, LLM Gateway charges as other products do, but the setup time is quick, under a minute, with no cost for the free tier. After using LLM Gateway, we see that our security risks have reduced. In my organization, we do not only look at ROI; we also consider security threats. Since adopting LLM Gateway, the complexity of our projects has decreased, and the security concerns have lessened. I estimate it has saved us around 20-30% of our time, and around 30% sounds reasonable.

After using LLM Gateway, we see that our security risks have reduced. In my organization, we do not only look at ROI, but we also consider security threats. Since adopting LLM Gateway, the complexity of our projects has decreased, and the security concerns have lessened. I estimate it has saved us around 20-30% of our time, and around 30% sounds reasonable.

I usually start with the free tier, which is very good. For the enterprise version, LLM Gateway charges as other products do, but the setup time is quick, under a minute, with no cost for the free tier.

I have not evaluated other options. The setup cost, time, and the free tier availability made LLM Gateway an easy choice for us.

The accuracy of LLM Gateway's output is quite good. If one model is down, it automatically redirects requests to another model, which is a very beneficial feature. My advice for others looking into using LLM Gateway is to start with it. It takes very little time to set up and has a user-friendly dashboard that displays model usage. You can also set thresholds, specifying the number of tokens or costs for each model, which is very convenient. Depending on the product size, since it is new in the market, our usage has been satisfactory. I have given this review a rating of 9 out of 10.