European online retailer · Retail, E-commerce & Consumer
AgentOps keeps a retailer's support agents above 90 % resolution as volumes triple
Continuous evaluation, drift detection, and cost routing kept resolution quality stable through peak season while inference spend per ticket fell 38 %.
Challenge
The retailer's support agents resolved 78 % of tickets in a September pilot; by Black Friday, quality had drifted under new product lines and returns policies, and inference spend was rising faster than ticket volume.
Approach
AgentOps onboarding instrumented every run with traces, online evaluation sampling, and cost attribution. Drift detection flagged new intents weekly; policy changes were pushed through a versioned knowledge base with regression tests.
Solution
A monthly operating rhythm: evaluation review, guardrail tuning, model routing changes, and a scorecard to the COO. Routing simple intents to a small model and reserving the large model for exceptions cut cost per ticket by 38 % with no quality loss.
Results
Resolved without human handoff (peak)
Inference cost per ticket
Ticket volume vs. pilot
P1 incidents during peak
“Peak season used to mean hiring 200 temps. This year it meant a weekly review meeting.”