A short list of metrics that actually predict business value, and a longer list of vanity metrics worth ignoring. Part of the Chat agents section of the TrainYourAgent documentation, written to be executed against rather than skimmed.
The vast majority of chat agent decisions can be made from four numbers. Containment rate is the percentage of conversations the agent resolved end to end without a human handoff. Conversion rate is the percentage of conversations that ended in the desired outcome — a demo booked, an order placed, a ticket filed, whatever success looks like for that surface. Time to resolution is the median number of turns from the customer's first message to…
For a well-built chat agent on a healthy knowledge base, expect containment rates between sixty and eighty percent in the first month, climbing toward eighty-five percent after a few rounds of tuning. Expect conversion rates that are at least equal to your existing human-staffed chat baseline, and frequently double it because the agent is available around the clock. Expect median time to resolution under three turns for simple lookups and under eight turns for booking…
Total messages handled is a vanity metric. It rewards a chatty agent over a helpful one. Intent recognition accuracy in isolation is a vanity metric — it does not matter that the agent correctly classified the intent if it then failed to act on it. Token spend per conversation is interesting for cost forecasting but tells you nothing about quality. Average session duration is genuinely misleading — both very short sessions and very long sessions…
Containment looks simple but has nuance. A conversation that ends because the customer gave up and closed the tab is not a contained conversation, even though no human was involved. We instrument the agent to detect drop-off patterns — sudden silences after a question, repeated rephrasing of the same ask, explicit frustration language — and exclude those from the containment numerator. The honest containment number is usually five to ten percent lower than the naive…
This article is part of the Chat agents section. A short list of metrics that actually predict business value, and a longer list of vanity metrics worth ignoring.