Inference economics: why tiered model routing is the most important architecture decision you will make

Agents run continuously, and every step is an inference call. Without a routing strategy, cost scales with volume and the large model does all the work — including the 80 % of steps a small model handles just as well.

The gateway pattern

Put every model behind one gateway. Each route declares the allowed models, a cost ceiling, PII handling, and a fallback. Applications ask for a capability, not a model.

Tiering

Classify steps by difficulty. Extraction, classification, and formatting go to a small open-weight model, often on your own GPUs. Reasoning and synthesis go to the large model. Measure quality per tier with your evaluation suite.

Results we see

In production systems, tiered routing typically cuts inference cost by 35–60 % with no measurable quality loss, and gives the CFO a cost per unit of work they can compare with the benefit.