Caching, model routing, context pruning, batch APIs, and 14 other knobs that took our cost-per-conversation from 41 cents to 9 cents.
Our cost per conversation went from 41 cents to 9 cents in 90 days without changing the model. Below: each knob, the impact, and the order to pull them in. The failure modes we have learned to engineer around Five failure modes show up in this category over and over. A prompt that worked in week one starts producing off-brand replies by week four because the model behind it silently versioned. Fix: pin the model version, run a weekly voice-drift eval against 50 canonical scenarios, alert on a 3-point deviation.
A prompt that worked in week one starts producing off-brand replies by week four because the model behind it silently versioned. Fix: pin the model version, run a weekly voice-drift eval against 50 canonical scenarios, alert on a 3-point deviation. The KB updated, the embeddings did not. The agent confidently quotes last quarter's pricing. Fix: a freshness-check job that compares KB modified timestamps against embedding job runs hourly, and a hard ceiling that prevents serving any answer grounded in a document older than the freshness window.
Caching, model routing, context pruning, batch APIs, and 14 other knobs that took our cost-per-conversation from 41 cents to 9 cents. It is filed under AI Infrastructure because that is where operators looking for this problem actually start, and it is written from production work rather than from a content calendar.