Production Self-Hosted LLM Gateway: LiteLLM vs Ollama vs vLLM vs LocalAI (2026 Benchmark, TCO & Deployment Blueprint)

Bottom Line: LiteLLM is a gateway (+2 ms median / +13 ms p99 at 1,170 RPS on 4 small instances); vLLM, Ollama and LocalAI are engines. On one A100-40GB with Llama-3.1-8B FP16, vLLM peaks at 793 tok/s / 80 ms p99 versus default Ollama at 41 tok/s / 673 ms p99 — a 19× gap that appears only under concurrency. A saturated single-H100 vLLM node serving Llama-3.3-70B FP8 costs $0.17–0.81 per 1M output tokens in GPU rent ($0.5–1.75 after a 3–5× ops multiplier), at or below hosted Llama-3.3-70B APIs ($0.10–1.04) and 12–60× below GPT-5 output ($10.00); the same node at 1 user costs $3.45–16.16 per 1M. Utilization, not engine choice, dominates TCO. Reference stack: LiteLLM → vLLM, with Prometheus, Postgres and Redis. No defensible LocalAI concurrency or CPU-only cost-per-token figure exists as of 2026-09-16.

Scope and method #

Every figure below carries a source, hardware and date, compiled 2026-09-16, tagged [P] primary (vendor docs or method-disclosed benchmark), [S] secondary (aggregator, method not re-run) or [U] unverified (no primary source; caveated). Where sources conflict both values are shown. Derived cost figures are calculated from cited inputs and labelled as such.

The four tools are two categories #

vLLM, Ollama and LocalAI execute model weights. LiteLLM runs no models; it proxies 100+ upstream providers behind one OpenAI-compatible schema. Engines are measured in tokens per second, a gateway in requests per second plus added latency, and a production deployment stacks them: gateway → engine(s).

Throughput and latency benchmarks #

vLLM #

The one primary-source head-to-head is Red Hat Developer's run (published 2025-08-08, updated 2026-07-13): Llama-3.1-8B-Instruct FP16 on 1× A100-PCIe-40GB, CUDA 12.4, OpenShift 4.17, vLLM 0.9.1, load generated by GuideLLM 0.2.1 sweeping 1→256 concurrent users for 300 s per run [P].

• Peak throughput: 793 tok/s.
• P99 latency at peak: 80 ms.

A secondary 2026 single-run from Spheron on a larger model — Llama-3.3-70B-Instruct FP8, 1× H100 SXM5 80GB bare metal, vLLM v0.18.0, 200 prompts averaging 512 input / 256 output tokens — gives the scaling curve [S]:

Cold start on that configuration was ~62 s [S]. Caveat: the Spheron figures are blog-grade, single-run, with no raw data published; they are corroborated in shape (near-linear scaling with concurrency) but not in exact value by the Red Hat primary run.

Ollama #

Same Red Hat rig, same model at llama3.1:8b-instruct-fp16, Ollama 0.9.2 [P]:

• Peak throughput: 41 tok/s (derived ratio: 19.3× below vLLM's 793 tok/s).
• P99 latency at peak: 673 ms (derived ratio: 8.4× vLLM's 80 ms).

The gap is a configuration consequence, not only an engine one. Ollama's OLLAMA_NUM_PARALLEL defaults to 1 and queued requests are processed in FIFO order [P: Ollama FAQ]. Raising it narrows the gap, but memory scales linearly with parallel slots × context length [P: Ollama FAQ], and no primary benchmark of tuned Ollama at high concurrency was found [U] — the 19× figure applies to default settings only.

At a single user the picture inverts. Secondary sources report Ollama TTFT of ~45 ms versus vLLM ~82 ms on Llama-3.1-8B [S: Markaicode], and SitePoint's 2026 comparison puts Ollama 18% faster on single requests while vLLM delivers 2.3× the throughput at 8 concurrent [S]. At 50 concurrent, Markaicode measured Ollama plateauing at ~155 tok/s total with ~3,200 ms TTFT versus vLLM ~920 tok/s and ~145 ms TTFT [S — hardware undisclosed; prefer the Red Hat rows where both exist].

LocalAI #

No independent, method-disclosed LocalAI server throughput or concurrency benchmark was found [U]. LocalAI's default backend is llama.cpp and it can also load a vLLM backend [P: LocalAI features], so its ceiling is approximately the chosen backend's ceiling minus an unmeasured gRPC-hop overhead. The closest proxies are single-stream llama.cpp figures from Presenc AI (2026) [S]: 130–150 tok/s for 7B Q4 on an RTX 5090, 14–22 tok/s for 70B Q4 on the same card (offloaded, >32 GB VRAM), 35–45 tok/s for 70B Q4 on a DGX Spark, and 95–110 tok/s for 7B Q4 on a Mac Studio M5 Max 128GB via MLX. None of these measures LocalAI itself; state any LocalAI throughput as "≈ backend performance, overhead unmeasured."

LiteLLM #

LiteLLM's published benchmarks on 4 instances × (4 vCPU, 8 GB), v1.79.1-stable [P]: added overhead 2 ms median / 8 ms p95 / 13 ms p99; 1,170 RPS sustained at 100 ms median / 150 ms p95 / 240 ms p99 end-to-end. Separately (version not stated), the Realtime API reached 1,207 RPS at 59 / 67 / 99 ms on 4 instances with 1,000 users [P]. A high-throughput profile on v1.101.0 (33 pods × 4 workers, 528 GiB requested) reached 3,000 RPS at p50 30.6 / p95 54.0 / p99 91.6 ms [P].

An independent stress test (agentgateway, 2026-06-26, Docker with undisclosed host specs, 18 workers, Fortio 32 connections, 1 KB payloads) measured p50 7.08 ms / p90 17.99 ms / p99 32.19 ms at a 3,198 QPS peak with 11.8 GB peak RSS [S]. The memory figure is the operational takeaway: LiteLLM's Python hot path is not free at multi-thousand QPS.

A vendor-internal Rust hot-path rewrite is reported at ~0.05 ms overhead versus ~7.5 ms Python and 6,782 versus 453 req/s at 50 clients [U — pre-GA, 2026-06-22]. Do not size capacity on these numbers.

Comparison table #

Throughput and latency rows use the highest-confidence source available per tool. $/1M output token figures are derived in the TCO section below.

Trade-offs per tool #

LiteLLM #

• Advantage: the only component in the set with per-key budgets, team-scoped rate limits, cross-provider fallback and spend accounting [P]. Overhead is 2 ms median at 1,170 RPS [P].
• Drawback: two stateful dependencies (Postgres ≥12, Redis) for multi-instance use [P], 1 vCPU + 4 Gi per pod [P], 11.8 GB RSS at 3.2k QPS in an independent test [S], and SSO/audit behind an enterprise licence [P]. Throughput scales with pod count, so it is a recurring cost line.

vLLM #

• Advantage: highest measured throughput in the set — 793 tok/s at 80 ms p99 on an A100-40GB [P]; 2,400 tok/s at 100 concurrent on a 70B FP8 H100 model [S]. Native Prometheus metrics [P].
• Drawback: GPU-only; ~62 s cold start on 70B FP8 [S]; 76–78 GB VRAM at 50–100 concurrent leaves no headroom on an 80 GB card [S]. Single shared API key and no rate limiting [U/P] make it not suitable for direct multi-tenant exposure. Quantization list and TP/PP flags were not confirmed on the fetched docs page [U].

Ollama #

• Advantage: fastest single-user TTFT in the set (~45 ms vs vLLM ~82 ms) [S], runs on CPU or GPU, MIT-licensed, zero stateful dependencies.
• Drawback: 41 tok/s and 673 ms p99 at default config [P], 19× below vLLM on identical hardware; no server auth, port 11434 open by default [P]; /metrics unconfirmed [U]. Not suitable for shared traffic without a tuned OLLAMA_NUM_PARALLEL (memory grows linearly [P]) and an authenticating reverse proxy.

LocalAI #

• Advantage: widest modality surface — chat, embeddings, images, STT/TTS, realtime WebSocket and function calling in one binary [P]; can host a vLLM backend for GPU throughput or llama.cpp for CPU; P2P federation across nodes [P]; MIT.
• Drawback: zero method-disclosed throughput or concurrency data [U]; API-key protection exists but per-key quotas and metrics are unconfirmed. Any LocalAI capacity plan needs an in-house load test before commitment.

Reference architecture #

The deployment that follows from the feature matrix places the gateway in front of one or more engines, with auth, quotas and spend accounting at the gateway and raw metrics scraped from both layers.

flowchart LR
    C[Clients / apps] -->|OpenAI schema, virtual key| G[LiteLLM proxy<br/>N pods x 1 vCPU / 4 Gi]
    G -->|route: local-70b| V1[vLLM node A<br/>1x H100, Llama-3.3-70B FP8]
    G -->|route: local-70b| V2[vLLM node B<br/>1x H100, Llama-3.3-70B FP8]
    G -->|route: small-cpu| O[Ollama or LocalAI<br/>CPU / small GPU, <=8B Q4]
    G -->|fallback| API[Hosted API<br/>Together / Bedrock / OpenAI]
    G --- R[(Redis<br/>router state, rate limits)]
    G --- P[(Postgres >= 12<br/>keys, spend logs)]
    V1 -->|/metrics| M[Prometheus]
    V2 -->|/metrics| M
    G -->|/metrics + OTEL| M
    M --> D[Grafana / alerting]
    G -->|callbacks| L[Langfuse / OTEL traces]

Three consequences: auth lives only at the gateway, because neither vLLM (single shared key [U]) nor Ollama (no auth [P]) should be reachable outside the private network; the engine tier scales by node with a ~62 s cold start [S] while the gateway tier scales by pod; and the hosted-API fallback is what lets the self-hosted tier run near saturation, since overflow leaves the box instead of queueing.

TCO #

Inputs #

GPU on-demand rates (verified 2026-08-20) [S]: H100 from $1.49/hr (Vast.ai floor) through $1.99 PCIe / $2.69 SXM community and $2.99 SXM5 Secure Cloud (RunPod), $3.99 (Lambda), $4.25 (CoreWeave), up to $6.98 (hyperscalers). A100 from $1.99/hr.

API baselines, $ per 1M tokens (Sept 2026): Llama-3.3-70B cheapest listed $0.10 in / $0.32 out [S]; DeepInfra $0.23–0.40 blended [S]; Together serverless $0.54 (early 2026) or $1.04 flat (Sept 2026) — sources conflict, quote the range [S]; AWS Bedrock $0.72 flat [S]; Groq $0.59 in / $0.79 out, moved to enterprise-only on 2026-08-26 [S]. GPT-5 $1.25 in / $10.00 out; GPT-5-mini $0.25 / $2.00; GPT-5-nano $0.05 / $0.40 [P: OpenAI pricing, 2026-08-21].

Derived self-hosted $/1M output tokens #

Formula: $/1M = hourly_rate ÷ (tok_per_s × 3600 ÷ 1e6). GPU rent only; these are calculations from the cited inputs, not published figures.

Readings: a saturated single-H100 vLLM 70B node sits at $0.17–0.81 per 1M, below or at parity with every hosted Llama-3.3-70B option ($0.10–1.04) and 12–60× below GPT-5 output at $10.00. The identical box at 1–10 concurrent costs $0.64–16.16 per 1M, worse than every open-weight API; the 20× spread between the 1-user and 100-user rows exceeds any engine-to-engine difference except Ollama's default-config penalty, which makes it 19× the cost of vLLM on the same A100 ($13.48 vs $0.70). Third-party corroboration: Mixtral 8x7B on H100 falls from $15.25 to $0.87 per 1M as load rises from 1 to 25 RPS [S: LeanLM]; "Llama 4 70B on H100 at batch=8 ≈ $0.18/1M output" [S: Spheron].

Ops overhead and break-even #

LeanLM's all-in multiplier over raw GPU rent is 3–5×, with a coverage ratio of one mid-level MLOps engineer per 4–6 GPUs [S]. Applied to the saturated-H100 row, effective cost becomes $0.5–1.75 per 1M, which is roughly at parity with Together ($0.54–1.04) and Bedrock ($0.72) and still well below frontier APIs — but only at saturation. SitePoint places break-even against a managed API at roughly 2–5 M tokens per day on reserved capacity over a 12-month window [S].

Add the gateway line: 1 vCPU + 4 Gi per LiteLLM pod plus Redis and Postgres [P]. At 3.2k QPS an independent test saw 11.8 GB RSS [S], so budget memory per pod, not only CPU.

CPU-only cost #

No 2026 source with disclosed method gives tok/s per CPU-dollar for a multi-user server-class CPU [U]. Single-stream consumer figures exist (Mac M5 Max, 7B Q4, 95–110 tok/s [S]) but carry no price. CPU hosting should be stated as viable for fewer than 5 concurrent users on ≤8B Q4 models, with no cost-per-token claim.

Deployment blueprint #

Sizing figures below come from the cited benchmarks; adjust after an in-house load test.

Step 1 — Engine tier #

• Model: Llama-3.3-70B-Instruct FP8 on 1× H100 80GB per node; expect 76–78 GB VRAM at 50–100 concurrent [S], leaving no room for a co-resident model.
• Set --max-num-seqs to the concurrency you have benchmarked (the Spheron curve suggests 50–100 on this card) [S]. vLLM has no rate limiting beyond this flag.
• Plan for a ~62 s cold start [S]: keep a warm standby or pre-pull weights before scaling events.
• Bind only to the private network. Verify the --api-key, quantization and tensor-parallel flags against docs.vllm.ai before launch — they were not confirmed on the fetched docs page [U].
• For a small-model or CPU pool, run Ollama with OLLAMA_NUM_PARALLEL raised from its default of 1 [P]; memory scales linearly with slots × context [P]. Never expose port 11434 directly [P].

Step 2 — Gateway tier #

• LiteLLM at 1 vCPU + 4 Gi per pod, 1 uvicorn worker per pod, scaled horizontally [P]. Four such instances sustained 1,170 RPS at 2 ms median overhead [P].
• Provision Postgres ≥12 and Redis; both are required for shared router state and rate limits across instances [P].
• Define one virtual key per team with RPM/TPM budgets [P]. Configure routing_strategy with the two vLLM nodes as primary and a hosted API as fallback.

Step 3 — Observability and auth #

• Scrape vLLM /metrics [P] and LiteLLM Prometheus metrics [P]; wire LiteLLM OTEL or Langfuse callbacks for traces [P]; spend logs land in Postgres [P].
• Do not rely on an Ollama /metrics endpoint: its existence is unconfirmed in official docs [U], and if present it is reportedly unauthenticated.
• SSO and audit logs on LiteLLM are enterprise-licensed [P]; budget for the licence or front the gateway with an external IdP.

Step 4 — Validate before commitment #

Run a 300 s sweep at 1→256 concurrent (the Red Hat method [P]) on your own node; published $/1M varies 20× with utilization on identical hardware, so only your own curve sets a defensible figure.

Final verdict by scenario #

Single-node development. Ollama. Its ~45 ms single-user TTFT [S], CPU/GPU flexibility and zero dependencies fit a one-person loop. Do not extrapolate its numbers to production: 41 tok/s and 673 ms p99 at default config [P].

Multi-GPU production. vLLM behind LiteLLM. vLLM is the only engine with primary-source throughput data (793 tok/s, 80 ms p99 [P]) and native metrics; LiteLLM supplies the auth, quotas and fallback that vLLM lacks. At saturation this stack lands at $0.17–0.81 per 1M raw, $0.5–1.75 all-in [derived], versus $10.00 for GPT-5 output [P]. Below roughly 2–5 M tokens/day [S], a hosted Llama-3.3-70B API at $0.10–1.04 [S] is cheaper.

CPU-only. Ollama or LocalAI on ≤8B Q4 models for fewer than 5 concurrent users. No cost-per-token figure can be defended from available sources [U]; treat this as a capability decision, not a cost one. Choose LocalAI over Ollama only if you need STT/TTS, images or realtime in the same binary [P].

Multi-provider routing. LiteLLM is the only tool in the set that does this [P]; the others are engines. Accept the Postgres + Redis dependency, 1 vCPU + 4 Gi per pod and up to 11.8 GB RSS at 3.2k QPS [S] as the price of per-key budgets and cross-provider fallback. Ignore the Rust hot-path numbers until GA [U].

Omitted by design. LocalAI concurrency throughput, tuned-Ollama high-concurrency throughput, and CPU-only $/1M — no source with a disclosed method exists for any of the three as of 2026-09-16.

Sources #

• Red Hat Developer — Ollama vs vLLM deep dive (2025-08-08, upd. 2026-07-13): https://developers.redhat.com/articles/2025/08/08/ollama-vs-vllm-deep-dive-performance-benchmarking
• Spheron — vLLM vs TensorRT-LLM vs SGLang H100 (2026): https://www.spheron.network/blog/vllm-vs-tensorrt-llm-vs-sglang-benchmarks/
• LiteLLM docs — Benchmarks: https://docs.litellm.ai/docs/benchmarks
• LiteLLM docs — Production: https://docs.litellm.ai/docs/proxy/prod
• Markaicode — LiteLLM production latency (2026): https://markaicode.com/benchmarks/litellm-production-benchmark-latency/
• Markaicode — Ollama vs vLLM (2026): https://markaicode.com/benchmarks/ollama-vs-vllm-performance/
• SitePoint — Ollama vs vLLM 2026: https://www.sitepoint.com/ollama-vs-vllm-performance-benchmark-2026/
• Ollama FAQ: https://docs.ollama.com/faq
• Ollama issue #3144 (metrics): https://github.com/ollama/ollama/issues/3144
• LocalAI features: https://localai.io/docs/features/
• vLLM online serving docs: https://docs.vllm.ai/en/latest/serving/online_serving/
• Presenc AI — local tok/s benchmarks 2026: https://presenc.ai/research/local-llm-tokens-per-second-benchmarks-2026
• IntuitionLabs — H100 rental prices (verified 2026-08-20): https://intuitionlabs.ai/articles/h100-rental-prices-cloud-comparison
• Spheron — RunPod H100 pricing 2026: https://www.spheron.network/blog/runpod-h100-pricing-2026/
• SynpixCloud — cloud GPU pricing 2026: https://www.synpixcloud.com/blog/cloud-gpu-pricing-comparison-2026
• OpenAI pricing (2026-08-21): https://developers.openai.com/api/docs/pricing
• PricePerToken — Llama 3.3 70B: https://pricepertoken.com/pricing-page/model/meta-llama-llama-3.3-70b-instruct
• Markaicode — Llama 3.3 70B API cost: https://markaicode.com/pricing/llama-33-pricing/
• CloudZero — Groq pricing 2026: https://www.cloudzero.com/blog/groq-pricing/
• AI Pricing Guru — Together pricing: https://www.aipricing.guru/together-pricing/
• LeanLM — self-hosting cost: https://leanlm.ai/blog/self-hosting-llm-cost
• SitePoint — self-hosted LLM costs 2026: https://www.sitepoint.com/self-hosted-llm-costs-2026/
• Spheron — GPT-6 vs self-hosted 2026: https://www.spheron.network/blog/gpt-6-vs-self-hosted-llm-2026/