Controlling LLM costs in production without hurting quality

Most teams “control LLM costs” by swapping their model for a cheaper one and hoping. That works until support tickets start asking why the assistant got dumber. Real cost control is an architecture, not a model swap: route each request to the right model, cache what repeats, and fall back gracefully when something breaks. Done right, none of it touches quality — because you’re removing waste, not capability.

1. Route by request, not by product

Not every message needs your frontier model. A classifier question, a summarization pass, or “what’s my order status” doesn’t need the same model as a multi-step reasoning task. Model routing sends each request to the cheapest model that can still answer it correctly:

• Cheap/fast tier — intent classification, extraction, short FAQ answers, formatting.
• Mid tier — general chat, drafting, most RAG answers.
• Frontier tier — multi-step reasoning, code generation, anything where a wrong answer is expensive to unwind.

The trap is optimizing only for token price. A downgraded model that causes a retry, a bad tool call, or a support escalation didn’t save you money — it moved the cost downstream and made it worse. Route on representative production traffic, score outputs against your actual task (not a generic benchmark), and only promote a cheaper model into a tier once it matches quality on that specific job. Re-check periodically — providers change models under stable names.

2. Cache aggressively — most of your prompt doesn’t change

If you’re not using prompt caching, you’re paying full price for the same system prompt, tool definitions, and RAG context on every single call. Anthropic prices cached reads at roughly a tenth of fresh input tokens; OpenAI auto-caches repeated prefixes at about half price with zero code changes. Teams that structure prompts around this routinely cut a large chunk of their bill without touching a single model choice.

The rule that makes it work: put what changes last.

[system prompt]        <- stable, cache this
[tool definitions]     <- stable, cache this
[retrieved context]    <- semi-stable per session, cache this
[user's latest message] <- changes every call, goes last

If you interleave stable and dynamic content, you invalidate the cache on every request and pay full price anyway. Long system prompts, big tool schemas, and repeated RAG context are exactly where caching pays off — the more boilerplate at the front of your prompt, the more this matters.

3. Fallbacks are a cost feature, not just an uptime feature

A fallback chain — primary model → secondary model → tertiary model — is usually framed as reliability engineering. It’s also a cost lever: a rate limit or outage on your primary provider shouldn’t mean paying panic prices for whatever’s left, and it shouldn’t mean silently serving worse answers without knowing it.

Two things make fallbacks safe instead of a quality leak:

1. Match capability, not just availability. Your fallback model should sit in the same quality tier as your primary for the tasks that actually reach it. A frontier model falling back to your cheap-tier model will produce visibly worse answers on exactly the requests that needed the frontier model in the first place.
2. Log every fallback event. If you don’t track how often you’re on the fallback path, you won’t notice when a provider degrades quietly, and you won’t know your real blended cost.

Put it together

These three pieces compound. Routing cuts the average cost per request. Caching cuts the cost of every request that still hits your best model. Fallbacks stop an outage from turning into either a quality collapse or a price spike. None of it requires degrading the answer a user actually gets — it requires being deliberate about which model, which tokens, and which path earn the cost of a frontier call.

We build this routing-and-caching layer into every AI product we ship, because “just use GPT for everything” is a launch decision, not a production one.