Keeping LLM Costs Under Control in Production

The first production bill is usually the surprise. A feature that cost pennies in testing costs real money once a few thousand people use it, and the reason is rarely a single expensive call — it is thousands of ordinary ones that nobody budgeted.

Know what you are paying for

Token cost is driven by input far more often than output. A system that attaches a long instruction block, several retrieved passages and a conversation history to every request is paying for all of it on every turn, regardless of how short the answer is.

Before optimising anything, instrument. Log input tokens, output tokens, model and latency per request, tagged by feature. Without that, cost reduction is guesswork, and teams routinely optimise the wrong call.

The levers, roughly in order of effect

Send less context

Retrieving twenty passages and sending all of them is the most common waste. Retrieve widely, rerank, send the best three or four. Quality usually improves at the same time — models attend worse to long, diluted context than to short, relevant context.

Route by difficulty

Not every request needs your most capable model. Classification, routing, extraction and short rewrites are often handled well by a smaller, cheaper model. Reserve the expensive one for the requests that genuinely need reasoning.

A cheap classifier deciding which model handles each request pays for itself almost immediately.

Cache

Three layers are worth having. Embedding caches, because the same text gets embedded repeatedly. Response caches for identical or normalised-identical queries. And prompt caching where your provider supports it, which makes a long shared instruction block much cheaper across calls.

Trim the conversation

Sending the entire history on every turn grows cost quadratically over a session. Summarise older turns, or keep a rolling window plus a running summary. Most conversations do not need turn three verbatim by turn twenty.

Set ceilings before you need them

Per-request token limits, per-user daily caps, and a global spend alarm. These are not optimisations, they are safety equipment. The scenarios that produce shocking bills are not gradual growth — they are an agent that loops, a retry storm, or a single user scripting against your endpoint.

Cap first, tune later.

Streaming changes the economics of patience

Streaming does not reduce cost, but it changes what users will tolerate. A response that takes six seconds to arrive whole feels broken; the same response streaming from 400ms feels immediate. That often removes pressure to switch to a faster, more expensive model for perceived-performance reasons.

The measurement that matters

Cost per successful outcome, not cost per call. A cheaper model that needs two attempts and still produces something a human has to fix is not cheaper. Track the full path — retries, fallbacks, human correction time — and the right model choice is usually obvious.