OpenCost 1.121.0 finally measures the real per-token cost of Kubernetes inference
On August 5, 2026, OpenCost teamed up with llm-d to ship the first per-token inference cost tracking for Kubernetes. Splitting allocation cost from usage cost ends the flawed math that was wrongly justifying self-hosted LLMs.
August 5, 2026. OpenCost, the CNCF incubating project, shipped version 1.121.0 with a first: per-token inference cost tracking on Kubernetes, through an integration with llm-d, the CNCF sandbox project for distributed LLM serving. The announcement, co-authored by Sima Nadler (IBM Research) and Alex Meijer (OpenCost maintainer), answers the question platform teams have been unable to answer for two years: what does a token produced by your own cluster actually cost?
The GPU bill keeps climbing, models serve billions of tokens, and the link between those two numbers is invisible. Three decisions are made blind today:
- Self-host or consume a SaaS API? You are guessing.
- Which model is actually cost-efficient at your traffic level? The data does not exist.
- Which team’s agent workload is eating the AI budget? Nobody knows.
The result is a large monthly bill with no explanation, while leadership asks ROI questions you cannot answer with numbers.
Why now
The question is not academic. The GPU bill is climbing while models serve billions of tokens, and platform teams still have no way to connect those two numbers. FinOps, which structured general cloud spend, simply did not cover AI inference: a billed GPU pod, yes; a token, no. In July 2026, the FinOps Foundation extended its framework to licenses, SaaS and data center — inference is the next frontier, and it badly lacked instrumentation.
Until now, teams compensated with back-of-the-envelope estimates: an average price per model, a rough tokens-per-hour ratio. Those approximations held while AI was a prototype. They no longer hold once production and chargeback arrive — which is exactly when OpenCost enters the picture.
Cost ≠ price
The post opens with a clarification: cost is not price. A SaaS provider may price below its own cost to buy market share, or far above for a premium model. When you compare self-hosting to SaaS, the SaaS cost for your enterprise is the sticker price — not its internal cost. The distinction sounds academic; it is the root of the most common modeling mistake.
Two costs, two questions
The core insight: there are two different questions, and they require two different metrics.
- Allocation-based cost includes everything it takes to keep a model available: the GPU memory reserved for its weights, the compute consumed during active inference, and a share of common infrastructure (gateway, KV cache storage). It reconciles with your infrastructure bill and answers “what is this model costing us?”.
- Usage-based cost counts only the compute consumed during active inference, crediting savings from KV cache hits. It answers “what did this model’s actual work cost?”.
The gap between the two is the cost of keeping the model warm. A low-traffic model can spend 95 % of its time in this “warm but idle” state: its weights occupy VRAM no matter what, burning budget while producing zero useful tokens.
The relationship between the two metrics directly expresses GPU utilization, with no separate metric:
# Utilization = usage-based / allocation-based
llm_cost_per_million_tokens{cost_basis="usage"}
/
llm_cost_per_million_tokens{cost_basis="allocation"}
# Example: $1.00 (usage) / $4.00 (allocation) = 25% utilizationThe build-vs-buy trap
The classic modeling error is justifying self-hosting with the usage-based cost. If your usage cost is $1.00 per million tokens and the SaaS API charges $2.00, self-hosting looks twice as cheap. But usage cost captures active compute only: it excludes GPU reservation, idle time and infrastructure. At 25 % utilization, the real self-hosting cost is $4.00 per million tokens, not $1.00.
The correct comparison is allocation-based cost vs. API price:
Self-hosted model at 25% utilization:
usage-based cost per million tokens : $1.00 (compute only — misleading)
allocation-based cost per million tokens : $4.00 (real cost — use this one)
external API price per million tokens : $2.00
Conclusion: the external API is cheaper at current utilization.
Self-hosting becomes competitive above ~50% utilization.The same framing yields the optimization target: raising utilization through smarter routing, model sharing or traffic consolidation lowers the allocation cost per token — and can make self-hosting economical.
What the integration actually measures
The integration builds on metrics already present in an llm-d deployment: vLLM token throughput (vllm:prompt_tokens_total, vllm:generation_tokens_total), GPU costs from OpenCost’s allocation engine, and processing-time metrics. vLLM users who do not run llm-d benefit too, since the core metrics come from vLLM.
Two metrics are published to Prometheus and exposed through OpenCost’s REST API:
| Metric | What it measures |
|---|---|
llm_total_hourly_cost | Hourly cost per model |
llm_cost_per_million_tokens | Cost per million tokens, with labels splitting input and output |
Input and output token costs (including reasoning tokens) are reported separately because they exercise different phases: input drives the prefill, output drives the decode. In disaggregated serving — prefill and decode on separate hardware — this split is essential for fair attribution. The metrics also account for KV cache hits on input-token processing cost.
Beyond the GPU: the full cost of a model
A model served by llm-d does not run in isolation. The full picture includes the inference scheduler (EPP), the gateway proxy, KV cache storage — up to 18 TB in tiered deployments — and the Workload Variant Autoscaler. OpenCost distributes these shared costs through its SharedLabels mechanism, keeping the two projects decoupled: llm-d labels its components at deploy time, OpenCost attributes the costs without understanding llm-d’s internals.
Reading the cost matrix
The four combinations of allocation and usage cost each tell a distinct story:
| Allocation | Usage per million | Diagnosis |
|---|---|---|
| High | Low | Expensive to keep available, efficient during inference: the problem is utilization. |
| High | High | Expensive to host and to run: is this the right model? |
| Low | Low | Likely a well-sized deployment. |
| Low | High | Cheap to host, costly to run: evaluate size, quantization and hardware fit. |
A finance team can query costs by namespace and team label to produce showback reports. A FinOps team spots underutilized models and quantifies the savings from right-sizing or decommissioning. And a smart router — which llm-d is building — can factor per-token cost into routing decisions alongside latency and throughput.
The feature was validated on a cluster with 109 GPUs and 30 deployed models.
Deployment in practice
Enabling the feature does not mean rewriting your stack:
- Under llm-d, upgrading to OpenCost 1.121.0 is enough: metrics flow to Prometheus, and the REST API exposes
llm_total_hourly_costandllm_cost_per_million_tokens. - Under vLLM without llm-d, you get the same core metrics, since the calculation is built on
vllm:prompt_tokens_totalandvllm:generation_tokens_total. - A deployment guide for OpenCost + llm-d shipped with the release.
First reflex once the metrics are live: display the allocation / usage matrix for your models and flag the “high allocation, low usage” rows. Those are your warm-and-idle GPUs — the easiest money to recover through traffic consolidation or model sharing. OpenCost remains fully open source, so trying it costs a single helm install, not a procurement cycle.
Verdict
If you serve LLMs on Kubernetes, you cannot make a build-vs-buy call without the allocation-based cost per million tokens — and until now, no metric gave it to you. OpenCost 1.121.0 closes that gap.
The decision rule is now computable: compare allocation cost to the API price, never the usage cost. Below ~50 % GPU utilization, the external API almost always wins; above it, self-hosting becomes competitive. Before buying more GPUs, deploy OpenCost and read your cost matrix: the savings are usually in traffic consolidation, not in hardware.