Your LLM bill is a routing problem

Most AI features reach production the way the prototype was built: one model, the strongest available, and a long prompt sent in full on every call. That's right for a prototype: it takes model quality off the table while you learn whether the feature works. It's an expensive way to run. Every request, easy or hard, goes to the top-priced model, and you pay full price to resend the same instructions thousands of times a day.

What AI implementation actually costs covers the budget. This post is the engineering side: lowering the run cost of a live feature. Measure first, then four levers, biggest first.

What is LLM cost optimization?

LLM cost optimization is lowering the cost per completed task without lowering quality on your eval set. Cost per token is what the vendor prices. Cost per task is what you pay: every call, every retry, every token in and out. And without an eval set, you can't tell a saving from a quietly worse product.

How do you measure LLM cost per task?

Log six things per model call, tagged by feature:

Calls per task. Agents and retries multiply this quietly.
Stable input tokens: system prompt, tool definitions, reference documents.
Variable input tokens: the user's message, retrieved context, conversation history.
Output tokens. On Anthropic's price list in October 2026, output costs five times as much as input on every model.
The model each call used, so you can see the mix.
Who's waiting: a person who needs the answer in seconds, or nobody until tomorrow.

Price each count and sum across a task's calls: that's cost per task by feature. Put it on the dashboard next to quality and p95 latency, the three numbers from our pilot-to-production roadmap.

Don't compare models on price per token alone. Anthropic's pricing docs say its newer models use a tokenizer that produces about 30% more tokens for the same text. A lower price per token can still mean a higher cost per task.

Why is model routing the biggest lever?

Because the price spread between models is wider than any discount. On Anthropic's list in October 2026, input tokens on its most expensive current model cost 100 times what they cost on its least expensive one, for prompts up to 100,000 tokens. And yesterday's frontier is today's small tier: Stanford HAI's 2025 AI Index found that the inference cost of a system performing at GPT-3.5's level dropped more than 280-fold between November 2022 and October 2024.

Plenty of production traffic never needed the top model: tagging a ticket, pulling fields from an invoice, a yes-or-no check. Three ways to route, in order of effort:

Rules by task type. Each feature, or each step inside an agent, gets the smallest model that passes its evals. Start here. It's config in your gateway.
A learned router that sends each request to a strong or a weak model. The RouteLLM paper (Ong and colleagues, 2024) trained routers on human preference data and reported cost reductions of more than 2x in some cases without compromising response quality. Benchmark results, not your traffic.
A cascade. Try the small model, check the answer with a validator, escalate when it fails. The FrugalGPT paper (Chen, Zaharia and Zou, 2023) matched the best single model it tested, GPT-4, at up to 98% lower cost on its datasets. Proof of the idea, not a saving to plan on: those models and prices are gone.

What should you put in the prompt cache?

The stable part of the prompt. OpenAI and Anthropic both offer prompt caching: they store a prefix that's identical across calls and bill reused tokens at a fraction of the input price. Their terms in October 2026:

OpenAI caches by default on supported models, with cached input discounted by up to 95%. On its newest models the prefix needs at least 1,024 tokens, and a cache write costs 1.25 times the input price.
Anthropic caches when you ask it to. A 5-minute cache write costs 1.25 times the base input price, a 1-hour write 2 times, and a read 0.1 times on most models, less on some of its newest. By its docs' math, the 5-minute cache pays for itself after one read, the 1-hour cache after two.

The arithmetic on a hypothetical 10,000-token prompt with an 8,000-token stable prefix: at a 0.1x read rate, a warm call bills 800 token-equivalents for the prefix plus 2,000 for the rest. That's 2,800 instead of 10,000, or 72% less on input. Arithmetic, not a benchmark, and output doesn't move.

Caching rewards prompt discipline: instructions, tool definitions and reference material first, anything that changes per call last. Anthropic's docs list editing a tool definition among the changes that invalidate the whole cache, another reason to keep tool sets small and stable. Caching whole answers to repeated questions is a separate layer you build yourself; the RAG production guide covers it.

Which workloads belong in a batch?

Anything nobody's waiting for. As of October 2026, OpenAI's Batch API bills at half the synchronous price and returns results within 24 hours. Anthropic's Batch API also halves input and output prices, and its docs say batch and caching discounts stack. Think dataset classification, nightly summaries, CRM enrichment and backfills. Your eval runs are the easy first candidate: same model, same prompts, half the price.

When does distillation pay off?

Last, and only for a narrow task at high volume. The method, as OpenAI's fine-tuning guide lays it out: tune a prompt on a large model until it meets your eval criteria, keep the outputs that pass, and fine-tune a smaller model on them.

Check the platform first. OpenAI is winding down self-serve fine-tuning: per its deprecations page, new jobs are already closed to organizations that never fine-tuned or haven't run a fine-tuned model in 60 days, and close for the rest on January 6, 2027. Wherever you train, own the dataset (eval set plus approved outputs) so you can retrain elsewhere, open-weight models included.

How do you cut LLM cost without cutting quality?

Let the eval set decide. Caching and batching don't change which model answers or what it sees, so they can ship once the telemetry is in. Routing, distillation and prompt edits, including reordering for the cache, can change the answer, so each ships only if the lower-cost path passes the same evals as the expensive one. A saving that drops your eval pass rate isn't a saving. It's a product decision, and someone should make it on purpose.

Cost per task is shared work. Data builds the telemetry routing depends on: tokens, cache hit rate, model mix per feature. Build runs the gateway and caching layer. Advisory turns cost per task into pricing and margin decisions for the AI features you sell. Our AI practice owns the routing and the evals that gate every saving. Four practices, one cost-per-task number: that's why we'd scope it as one piece of work, not four hand-offs.