Inference cost
Also: LLM cost, cost per request
What is Inference cost?
Inference cost is the running expense of getting answers from a deployed model, driven by tokens processed, model tier, request volume and infrastructure, as opposed to the one-off cost of building the system.
What Inference cost means
Inference cost is what you pay each time the model runs. For hosted APIs it is tokens in and out multiplied by the model's rate; for self-hosted models it is GPU hours, utilisation and the engineering to keep them busy. It scales with traffic, prompt length, answer length and how many model calls each user action triggers, which for agents can be many.
The levers are well understood: shorter prompts and answers, semantic caching for repeated questions, multi-model routing so simple requests hit cheap models, batching for background work, and prompt caching where the provider offers it. Measuring cost per conversation, per resolved ticket or per account turns the bill into a business metric rather than a line item.
Inference cost is not the total cost of ownership; evals, monitoring, retrieval infrastructure and people are separate. It is also not fixed: model prices fall over time, but usage grows and providers retire models, so a forecast needs revisiting each quarter.
Who it really matters to
- CFO: this is the variable cost of an AI feature and determines whether unit economics work at scale.
- Founder / CEO: pricing an AI feature depends on knowing the cost per account, not the average.
- CTO / Head of Engineering: architecture choices (routing, caching, context size) move this number by large factors.
- Product manager: feature design, like how chatty an assistant is, directly changes the cost per user.
Why it exists
The term exists because AI features break the usual software cost model: the marginal cost of serving one more request is real and visible, unlike most SaaS where infrastructure is a rounding error. Teams that ignore it launch features with negative margin at scale. Naming and tracking inference cost lets you design for it: route, cache, trim, and price accordingly. The trade-off is that every cost reduction can affect quality, so cost work must be evaluated alongside accuracy. Cheapest is not the goal; cheapest that clears the eval threshold is.
Where it is applied
- Reporting AI cost per account in a B2B SaaS so pricing tiers and usage caps reflect real margin.
- Cutting support-agent spend by routing FAQ-type questions to a small model and reserving large models for complex cases.
- Comparing hosted API cost against a self-hosted open-weight deployment for a bank that needs data to stay on-premise.
- Budgeting a voice agent for a hospital by combining per-minute telephony, speech and model costs.
- Caching answers to recurring student queries during admissions season to flatten a traffic spike.
Is Inference cost a skill?
MetricA number you track per request, per conversation and per account. Eazyware builds cost reporting into LLM applications and uses model-agnostic routing so the metric can be managed without rewriting the product.
Eazyware service that covers it: LLM Application Development. Starting prices are on the pricing page.
Frequently asked questions
How do we forecast inference cost before launch?
Measure real token counts per request type on a pilot, multiply by expected volumes and model rates, and add a margin for agent loops and retries. Then track actual cost per conversation from day one and adjust the forecast quarterly.
Is self-hosting always cheaper than an API?
No. Self-hosting wins at high, steady volume or when data residency demands it; at low or spiky volume, idle GPUs cost more than API calls. The comparison should include operations effort, not just hardware.