azyware
Technology

LLM inference costs: how to forecast your monthly bill

EZ
Eazyware
· 7 min read
Quick answer

How do you forecast LLM API costs for a production system?

Forecast from requests per day, tokens per request and model mix; routing and caching usually cut the first estimate by a third or more. Measure real prompts in shadow mode before committing to a number, separate input from output tokens, and revisit the forecast monthly, because model prices and usage both move.

LLM API cost forecasting comes down to three inputs: how many requests you make per day, how many tokens each request consumes in and out, and which models handle which share of the traffic. Multiply, add a margin for retries and tool calls, and you have a first estimate. Then apply routing and caching, which usually take a third or more off that first number. This article gives the formula, the mistakes that make forecasts wrong by a large multiple, and a way to turn a guess into a measured number before you sign anything.

Why the first estimate is usually wrong

Most teams estimate with the price of one model and the length of one prompt. Production is not one prompt. A single user question in an agent might trigger a classification call, a retrieval call, two tool calls with their results fed back, and a final answer, each with the system prompt repeated. Output tokens cost several times more than input tokens on every major provider, and agents produce more output than chat. The forecast has to model the whole chain per request, not the visible message.

The forecasting formula

Monthly cost equals the sum over each model of: requests per day multiplied by calls per request multiplied by (input tokens per call multiplied by input price plus output tokens per call multiplied by output price), multiplied by days in the month. The cleanest way to hold it is a small table with one row per call type, which is how we build every AI running cost estimate for a client.

InputWhere to get itCommon mistake
Requests per dayCurrent ticket, query or session volume, with peak daysUsing the average and forgetting month-end or campaign spikes
Calls per requestTrace one real request end to endCounting one call when an agent makes four
Input tokens per callMeasure with the provider's tokeniser on real promptsForgetting the system prompt, retrieved passages and tool results
Output tokens per callMeasure from shadow-mode logsAssuming output is short; agents reason and explain
Model mixDecide which call types need the frontier modelSending every call to the most expensive model
Retries and errorsProvider error rates plus your own timeoutsAssuming zero
Cache hit rateMeasure repeated prefixes and repeated questionsAssuming zero, or assuming near-total

Input tokens: the hidden multiplier

The system prompt is sent on every call. A retrieval step adds several passages. An agent's tool results are fed back as input. A long conversation resends the history each turn. A user who typed forty words can generate several thousand input tokens per call, and if the request makes four calls, the input side dominates the bill even though input is cheaper per token. When you measure, measure the full payload the provider sees, not the user message. Prices are published; OpenAI's pricing page is the reference for an OpenAI cost estimate and other providers publish equivalents.

Model mix: where routing earns its keep

Not every call needs the strongest model. Classifying an intent, extracting a field, deciding whether to escalate and summarising a tool result are tasks a small model handles at a fraction of the cost. The frontier model is reserved for the final answer, the ambiguous case or the complex reasoning step. Model-agnostic routing across OpenAI, Anthropic, Google and open-weight models lets you set that mix per call type and change it when prices or quality move. The mechanics are in multi-model routing. In the forecast, give each call type its own model and price rather than a blended rate.

Caching: two kinds, both worth modelling

Prompt caching, which the major providers now offer, discounts the repeated prefix of a prompt, which for most applications is the system prompt plus the stable context. Anthropic's prompt caching documentation describes how the discount works on their models; other providers have equivalents. Response caching is your own: if the same question arrives many times, serve the stored answer. Model both as a hit rate applied to the relevant token share. Be conservative until you have measured; cache hit rates in a demo are higher than in production because demo users ask the same things.

Turn the guess into a measurement

The forecast becomes reliable once the system runs in shadow mode on real traffic. Every call is traced with its model, input and output tokens and cost, and the dashboard shows cost per request by call type. Two weeks of shadow logs give you a distribution rather than an average, including the long tail of expensive conversations that skew the bill. We install this tracing in the first week of every build; the approach is described in LLM observability. A token cost calculator is a fine start; a trace is what you budget from.

A worked example

A field-service SaaS wanted a copilot inside their product to answer technician questions from manuals and job history. The first estimate used one frontier model and the average question length, and came out alarming. Tracing a real request showed a retrieval call, a short classification, and a final answer with several manual passages as context. Routing the classification to a small model, caching the long system prompt, and trimming the retrieved passages to the sections that mattered brought the per-request figure down substantially. The copilot then went into shadow mode for two weeks, and the measured cost per active user became the number in the pricing model for the feature. The in-app copilot case study describes the build.

Controls that keep the bill inside the forecast

  • A hard budget per day per environment, with alerts at thresholds
  • A per-user or per-tenant cap so one heavy user cannot drain the month
  • Maximum output tokens set per call type
  • Conversation history trimmed or summarised beyond a set length
  • Retrieval limited to the top passages, not whole documents
  • A cost column in the trace so any regression shows within a day
  • A monthly review of price changes across providers and the routing table

Forecast for growth, not for launch

Launch traffic is low. Build the forecast for the volume you expect in six months, because that is the number the pricing of your product or the budget of your support team depends on. Per-request cost should fall over time as routing and caching mature; total cost rises with adoption. Report both to finance, and report cost per resolved conversation or per active user rather than the raw bill, because that is the number that says whether the system is worth it. The wider view, including build and care, is in total cost of ownership for AI systems.

Team and timeline

A first forecast takes a day with your volume data and a traced request. A measured forecast takes two weeks of shadow mode with tracing in place. Both are part of every LLM application build, which starts at $21,000 / ₹13.6L, and of every agent build. If you want the forecast before you commit, a ProofRun POC sprint over three weeks produces a working slice with a measured cost per request alongside the accuracy evals. Ongoing cost tuning, including moving call types between models as prices change, is part of a Care Plan. Build prices are on the pricing page.

Before you start: a checklist

  • Get requests per day for a normal week and the busiest week of the year
  • Trace one real request and count every model call in the chain
  • Measure input and output tokens with the provider's tokeniser, including system prompt and context
  • Decide which call types need the frontier model and which do not
  • Estimate cache hit rate conservatively and plan to measure it
  • Set daily and per-tenant budget caps before launch
  • Agree the number finance will track: cost per request, per user or per resolution
  • Diarise a monthly review of provider prices and the routing table

Glossary

  • Token: the unit providers bill on; roughly three-quarters of a word in English, more for other languages
  • Input tokens: everything sent to the model, including system prompt, history, retrieved passages and tool results
  • Output tokens: what the model generates; priced higher than input
  • Routing: sending each call type to the model that handles it well at the lowest cost
  • Prompt caching: a provider discount for repeated prompt prefixes
  • Shadow mode: running on real traffic without acting, to measure accuracy and cost
  • Cost per resolution: total running cost divided by outcomes achieved

Questions clients ask

  • Should we self-host to cut inference cost? Only at high, steady volume or where data residency demands it. Below that, API pricing with routing is cheaper once you count GPUs and the people to run them.
  • What happens when a provider changes prices? The routing table is updated and the eval suite confirms quality holds. That is why the forecast is reviewed monthly, not set once.
  • Can we cap what a single customer costs us? Yes. Per-tenant budgets and output limits are standard controls and should be in place before launch.
  • Is the forecast part of the fixed price? Yes. Every build includes tracing and a measured cost per request from shadow mode, handed over with the code.

See multi-model routing, how much does AI development cost in 2026 and total cost of ownership for AI systems. Provider prices change; check OpenAI's pricing page and the equivalents before each review.

Forecast from traced requests, route by call type, cache what repeats, and budget for the traffic you expect rather than the traffic you have.

Frequently asked questions

How accurate is a first LLM cost forecast?

▾

Usually within a factor of two or three, because calls per request and input tokens are underestimated. Two weeks of shadow mode with tracing turns it into a number you can budget from.

Does routing to cheaper models hurt quality?

▾

Not when the eval suite decides which call types move. Classification, extraction and summarisation typically move without loss; the final answer often stays on a stronger model.

How much should we budget for AI running cost?

▾

Model it per call type and per expected volume rather than a rule of thumb. For most products it is far smaller than the build and care costs listed on the pricing page.