The cost of unmanaged AI: tokens, latency and the bill nobody forecast
LLM features fail economically before they fail technically. Token accounting, caching discipline and right-sized models keep AI features inside budgets that survive their first invoice.
Peter Njoroge
Director of Platform Engineering
AI features rarely die because the model stopped working. They die when the finance team sees the third invoice. The engineering discipline that prevents this is unglamorous and learnable.
Know your unit economics
Every AI operation has a price per call: input tokens times rate, plus output tokens times rate, plus latency converted into user patience. Log all of it per request - operation type, provider, model, token counts, duration. Aggregate weekly. A feature burning more in tokens than the revenue it touches is a decision to make consciously, not a surprise to absorb.
Cost per operation varies enormously by task class. Summarisation of a two-thousand-word article and tag suggestion on a headline differ by orders of magnitude. Price them separately and budget them separately.
Match model size to task difficulty
Classification and extraction tasks routinely succeed with small, fast models; open-ended drafting benefits from larger ones. Routing tasks to the cheapest model that meets a measured quality bar cuts spend sharply without visible quality loss. This requires evaluation harnesses that score outputs per task class - which you should have anyway.
Cache aggressively, invalidate honestly
Editorial content changes on human timescales. Summaries, embeddings and tag suggestions for a published article are stable until the article changes. Cache keyed on content hash and revision, invalidate on edit. Hit rates above eighty percent are common for content workloads and transform the economics.
Set circuit breakers
Rate limits, concurrency caps and per-day budget ceilings belong in code, not in procurement policy. When a bug causes a loop of expensive calls - and someday it will - the breaker turns a potential four-figure afternoon into a log entry. Pair the breaker with alerting so a tripped limit triggers investigation rather than silent feature degradation.