LLM Cost Control: Caching Strategies That Actually Cut the Bill

Token bills grow with usage, and unlike servers you cannot right-size them after the fact. Exact-match caching, provider prompt caching, and honest measurement, in that order.

LLM cost has an unusual shape: every call costs real money, the same question costs the same every time you ask it, and nobody notices the bill until it is a line item. The good news is that AI workloads are extremely repetitive, which makes caching the highest-leverage optimization available, usually worth more than switching models.

Measure before you optimize

Log tokens in, tokens out, model, and feature name for every call from day one. Cost per feature is the number that drives every decision that follows; without it you are guessing which feature to optimize and cannot prove any fix worked.

Exact-match caching: the boring 80 percent

Hash the normalized request (model, prompt, temperature, context) and store the response. Identical request, cached answer, zero tokens. It sounds too simple to matter until you look at real traffic: the same documents get summarized repeatedly, the same questions get asked by different users, and background jobs re-process unchanged inputs constantly.

const key = sha256(JSON.stringify({ model, system, prompt }));

const cached = await redis.get(key);
if (cached) return JSON.parse(cached);

const result = await callModel({ model, system, prompt });
await redis.set(key, JSON.stringify(result), { EX: 60 * 60 * 24 });
return result;

Provider prompt caching: pay less for the repeated prefix

Most real prompts are a large static prefix (system prompt, instructions, shared documents) plus a small dynamic tail. Providers now let you mark that prefix as cacheable and charge a fraction of the normal input price on cache hits. If your system prompt or context block is thousands of tokens and reused across requests, this is close to free money; structure prompts so the stable part comes first and the variable part last.

Set TTLs by how wrong a stale answer can be

• Deterministic transforms (extraction, classification of an unchanged document): cache for days or until the source changes.
• Content generation over slow-moving data: hours are usually safe.
• Anything touching live data (prices, availability, account state): do not cache the model answer; cache the expensive retrieval instead.
• Invalidate by source: when a document updates, delete its cache entries instead of waiting for expiry.

Takeaways

• Per-feature token logging comes first; it is the meter for everything else.
• Exact-match caching is trivial to build and pays for itself immediately.
• Order your prompts static-first to exploit provider prompt caching.
• Choose TTLs by the cost of staleness, not by habit.