senn-techsenn-tech
AI Infrastructure
AI Infrastructure2026-10-18· By Franz Senn

Bifrost vs LiteLLM: an open-source AI gateway and its 50x speed claim

The newsletter Alles nur geCloud (issue 254, 18 October 2026) lists Bifrost among its link tips: an open-source gateway with one OpenAI-compatible API for all providers. The author lists it as a competitor to LiteLLM and passes on that, by its own website, it is 50 times faster. We run LiteLLM as the gateway in front of our local models, so we looked at what stands behind that number.

GitHub logo
Data source: the maximhq/bifrost repository page on GitHub, retrieved 20 October 2026. All performance figures are vendor claims. (Quelle: Simple Icons, file github.svg (CC0))

What Bifrost does

The repo describes an AI gateway that can run as an LLM, MCP or agent gateway. The licence is Apache 2.0. The page does not name the implementation language outright, but the Go install command and the file core/bifrost.go point to Go.

The feature list reads like that of any current gateway:

  • One OpenAI-compatible API for more than 23 providers, among them OpenAI, Anthropic, AWS Bedrock and Google Vertex.
  • Automatic fallbacks between providers and models, load balancing across API keys, semantic caching.
  • Usage tracking, rate limits, access control, and budgets through virtual keys, teams and customers.
  • A web UI, an API or file-based configuration; for enterprises, OIDC with directory sync, Prometheus metrics, tracing and clustering.
  • A drop-in for the OpenAI, Anthropic and Google SDKs by changing the base URL.

To start it, the page gives npx -y @maximhq/bifrost or docker run -p 8080:8080 maximhq/bifrost, with the UI on port 8080.

What the GitHub page says about speed

The 50x claim is not on the GitHub page. There, Bifrost is said to add under 200 microseconds of overhead at 5,000 requests per second. The measurement table gives 59 microseconds on an AWS t3.medium and 11 microseconds on a t3.xlarge at 5,000 RPS, with a 100 percent success rate on both. LiteLLM shows up in the repo only as an SDK integration, with no comparison value. The same issue points out that the figures in the README and in the project description differ. The project website (getmaxim.ai/bifrost) does make the 50x claim, backs it with its own figures (P99 54 times faster at 500 RPS, 20 microseconds added latency) and elsewhere states 11 microseconds itself; these too are vendor numbers.

Gateway overhead at 5,000 requests per second (microseconds, vendor figures)t3.xlarge11 · 11 µst3.medium59 · 59 µsUpper bound in description200 · under 200 µs0200
Values from the performance table and the project description on GitHub. No measurement against LiteLLM is included. (Quelle: GitHub maximhq/bifrost)

The same table lists mean request latency including the provider: 2.12 seconds on the t3.medium and 1.61 seconds on the t3.xlarge. Against that, 59 microseconds is about one 36,000th. That is our own division of two vendor numbers. Even a gateway with many times the overhead would hardly show in daily use. Overhead starts to matter at very high throughput or with very short answers.

How to check it yourself

Whether 50x holds can only be answered with your own load, and the ground rules are simple: the same models behind both gateways, the same hardware, the same requests. A plan:

Gateway comparison with your own loadLoad profilerecord real requestsBoth gatewayssame host, same modelOverheadgateway time vs model timeFailuretest fallback and budgetOperationsauth, logs, updates
The last two steps usually decide more than latency does. (Quelle: senn-tech, our own recommendation)

What hurts in operation matters more than the microseconds: how safe the default configuration is, how reliably budgets and fallbacks kick in, what happens at an update. We have not tested Bifrost and make no recommendation. The numbers on its page are a reason to put it on the test list.

Further reading

Questions?
What is Bifrost?+

An open-source gateway under Apache 2.0 that exposes one OpenAI-compatible API for more than 23 providers and, per the repo, over 10,000 models. It adds automatic fallbacks, load balancing across API keys, rate limits, budgets through virtual keys, and an MCP gateway.

Is Bifrost really 50 times faster than LiteLLM?+

Alles nur geCloud, the newsletter, says the figure comes from the project's own website. The GitHub page does not contain it; it lists gateway overhead of 59 microseconds on a t3.medium and 11 microseconds on a t3.xlarge at 5,000 requests per second. Both are vendor numbers, and we have measured nothing independently.

Should we switch away from a LiteLLM gateway?+

A gateway's overhead is measured in microseconds, a model answer in seconds. Features, operations and security should decide, not the latency figure. A switch needs a test with your own load first.