The LLM Cost Optimization Checklist

Our cost per conversation went from 41 cents to 9 cents in 90 days without changing the model. We pulled 17 knobs. Below: each knob, the impact, and the order to pull them in.

The failure modes we have learned to engineer around

Five failure modes show up in this category over and over. Each has a specific fix.

1. Drift in prompt voice. A prompt that worked in week one starts producing off-brand replies by week four because the model behind it silently versioned. Fix: pin the model version, run a weekly voice-drift eval against 50 canonical scenarios, alert on a 3-point deviation.

2. Stale retrieval. The KB updated, the embeddings did not. The agent confidently quotes last quarter's pricing. Fix: a freshness-check job that compares KB modified timestamps against embedding job runs hourly, and a hard ceiling that prevents serving any answer grounded in a document older than the freshness window.

3. Quiet hallucinations. The agent invents a policy or a part number with high confidence. Fix: every customer-facing answer must cite at least one source from retrieval, and the eval set includes 25 adversarial questions designed to bait hallucinations. No source, no answer.

4. Escalation breakdown. The agent escalates to a human, but the handoff context is one sentence and the customer has to start over. Fix: structured handoff payload (intent + history + sentiment + suggested next action), and a human eval pass on every 50th handoff.

5. Silent integration failure. The CRM webhook 500s, the agent acts as if it succeeded, the customer thinks the appointment is booked. Fix: synchronous confirmation back to the agent before it tells the customer anything is done, and a retry queue with paging on persistent failures.

The 72-hour deploy plan

The build sprint below runs on a 72-hour clock. That is the engineering, not the engagement: our published promise is live in 21 days from kickoff, which wraps this sprint in scoping, evals, shadow mode and cutover. If you see "72 hours" and "21 days" on this site and wonder which is true, both are — one is the part where code gets written.

Hour 0-8. Kickoff. Interview the two people who do this job today. Pull 50 sample inputs (calls, chats, tickets). Establish baseline metrics. Identify the three top customer intents.

Hour 8-24. First-pass prompt. Wire the orchestration. Stand up the eval harness with 25 cases drawn from the sample inputs. The eval harness has to exist before the first prompt does.

Hour 24-40. Integrations. CRM webhook, calendar booking, payment link if relevant. Each integration ships with a synchronous confirmation path.

Hour 40-56. Internal QA. The two people we interviewed in hour 0 spend 90 minutes running the agent through their hardest scenarios. Their feedback drives the second-pass prompt.

Hour 56-68. Shadow traffic. Real customer interactions, AI answers, human reviews before the answer is sent. We are looking for any case where the AI's draft is worse than the human's draft.

Hour 68-72. Cutover. We flip the routing rule, monitor for the first hour, and hand off the on-call rotation to the client's champion. The implementer stays on standby for 7 days.

Operator note: The 72-hour clock is real but it assumes the client has decided on success criteria before we start. If success criteria are unclear at hour 0, the clock does not start until they are. This is the single biggest cause of pilot drift we see.

The 17 knobs, ranked by impact

1. Model routing. Haiku for 80% of intents, Sonnet for the 20% that need it. 50-65% cost savings.
2. Prompt caching. Anthropic's prompt cache. 80% off on the cached portion. Use it for system prompts and few-shots.
3. Context pruning. Drop conversation history older than N turns or older than X tokens.
4. Batch API for non-realtime. 50% off for outbound calls, summarization, eval. Worth it.
5. Response length caps. max_tokens defaults are too high; cap to actual need.
6. Stop sequences. Prevent over-generation. Often saves 30-50% on output tokens.
7. Function-call schema compression. Long schemas eat tokens; shorten descriptions.
8. RAG over context-stuffing. Retrieve 4 chunks not 40.
9. Skip the planner on simple intents. Match-and-respond for top-10 intents.
10. Embeddings caching. Cache embeddings for stable content; only re-embed on change.
11. Streaming. Same cost, better UX, sometimes cheaper if you cut off mid-stream.
12. TTL on session state. Stale sessions cost storage; expire them.
13. Smaller embedding dimension. 1024-dim vs 3072-dim if your eval allows.
14. Re-ranker for fewer chunks. Send 4 high-quality chunks not 12 mediocre.
15. Cost dashboards per intent. Spot the outliers fast.
16. Per-tenant rate limits. Prevent one tenant from blowing the budget.
17. Yearly contract negotiation. Worth the 30 min on the phone.

We dropped from $0.41 to $0.09 per conversation. The first 6 items accounted for ~80% of the savings.

The tradeoffs we made and why

Every architecture choice in this category is a tradeoff. Here are the ones we have made consciously, and the alternative we did not pick.

We use Pipecat instead of building our own orchestration. The win is months of saved engineering. The cost is being a release behind on a few model integrations.

We use Anthropic as the default LLM with OpenAI failover, not the other way around. The win is consistently lower hallucination rates in our eval set. The cost is slightly higher cost-per-token at the equivalent model tier.

We run a custom eval harness instead of using a vendor product. The win is the eval set lives in the same Git repo as the prompts, so PRs that change prompts fail the build if they break an eval. The cost is we own the upkeep.

We use Twilio over a cheaper telephony provider. The win is concurrency ceiling and the depth of the diagnostic tools. The cost is roughly 18 percent more per minute.

We deploy on Fly.io, not Vercel. The win is real persistent connections and global edge POPs that survive long-lived voice sessions. The cost is more devops overhead.

These tradeoffs are not laws. They are starting points. If you tell us you have an existing GCP estate, we will adapt the stack. The principles do not move; the implementations do.

What we would do differently next time

Looking back at the last six deployments in this category, three things we would do differently:

Start the eval harness on day zero. We have always said this and we have always slipped it. The first time we shipped without a regression eval, we caught a prompt change that silently degraded conversion by 11 percent for two weeks before anyone noticed. Now we treat the eval harness as the first deliverable, before the first prompt.

Get the executive sponsor in the first user-acceptance session. Not the project manager, not the ops lead. The owner or the C-suite person whose name is on the budget. Their reaction to the first live test changes the trajectory of the project. Their feedback in week four is too late.

Document the human escalation paths before the AI ships. Every project we have shipped that did not have written escalation procedures had a moment in week two when an unexpected case hit, the AI escalated, and nobody knew who was supposed to handle it. Documenting the human side before the AI ships is half a day of work that prevents a week of fire-fighting.

If you want to talk through how any of this applies to your specific situation, grab a 20-minute call. We do not pitch on the call. If you would rather read more first, the docs and our comparisons cover most of the underlying technology choices in writing.

Closing

This post is the short version. The long version takes 90 minutes and a whiteboard. If you want the long version, grab a slot and bring your worst metric. We will work backward from there.

Or, if you are still in the read-and-think phase, our docs and comparisons pages have most of the answers in writing.