This is for CTOs and finance partners who run, or are about to run, AI features that call a paid model API. The problem is simple: model spend has no natural ceiling. A runaway loop or a retry storm can spend all night, a long conversation can do the same, and the first alert is often the bill.
Take a hypothetical, and to be clear it isn't a real incident. An agent retries in an unthrottled loop overnight. A tool it depends on starts returning errors at midnight, and the agent treats each error as a reason to try again, resending a longer history every time. Nothing crashes and no alarm fires. By morning it has made an enormous number of calls, and every one was billed.
We practice forward deployed engineering (FDE), where senior engineers work inside a client's team, and we build the controls below into production FinOps and resilience work. They're deterministic code around a non-deterministic model, the same stance as in our field guide to vibe-coded failure modes.
What stops an LLM bill from running away?
Four controls, all enforced in code: real-time spend counters with hard caps, tiered model routing, semantic caching where traffic repeats, and circuit breakers with a retry cap and dead-letter queue. A prompt asking the model to be economical is not a control.
| Control | Stops | Where it runs |
|---|---|---|
| Spend counters and caps | Unbounded total spend | Before every model call |
| Tiered routing | Using an expensive model for easy work | Router in front of the model |
| Semantic cache | Paying repeatedly for the same answer | Before routing |
| Breaker, retry cap, DLQ | Retry storms and poison jobs | Around every call and in the queue |
How do you build real-time spend counters?
Keep one atomic counter per scope (organization, user, session) in Redis. Before each model call, reserve the estimated cost across all scopes in one step. If any scope would pass its cap, refuse the call. After the response, reconcile the reservation with actual token usage.
The check and the increment must be a single atomic operation. If you read a counter in one command and increment it in another, two concurrent calls both pass the check. A Redis Lua script runs without interruption, so nothing can interleave with it. Store cost as an integer (for example micro-dollars) because INCRBY only accepts integers.
-- KEYS[1..3]: org, user, session counters (give them one hash tag for Redis Cluster)
-- ARGV[1]: estimated cost (integer)
-- ARGV[2..4]: caps for org, user, session
-- ARGV[5..7]: TTL in seconds for org, user, session counters
local cost = tonumber(ARGV[1])
for i = 1, 3 do
local current = tonumber(redis.call('GET', KEYS[i]) or '0')
if current + cost > tonumber(ARGV[i + 1]) then
return {0, i} -- blocked; i says which scope: 1 org, 2 user, 3 session
end
end
for i = 1, 3 do
redis.call('INCRBY', KEYS[i], cost)
if redis.call('TTL', KEYS[i]) < 0 then
redis.call('EXPIRE', KEYS[i], tonumber(ARGV[i + 4]))
end
end
return {1, 0}
Calling it from Node with ioredis looks like this:
const [allowed, scope] = await redis.eval(
RESERVE_LUA, 3,
'{org:42}:spend:day', '{org:42}:spend:user:7', '{org:42}:spend:session:abc',
estimatedCost, orgCap, userCap, sessionCap, 86400, 86400, 3600
)
if (allowed !== 1) throw new SpendCapExceeded(scope)
Estimate the worst case before the call: input tokens you are about to send plus the maximum output tokens you allow. After the call, apply the difference between estimate and actual with a second INCRBY (negative if you over-reserved). Cap the maximum output length on every request, or the estimate means nothing.
Decide in advance what happens if Redis is unavailable. For agents and batch jobs we fail closed: no counter, no call. Interactive features may fall back to a small local cap. Our post on Redis Lua versus database locks covers the atomicity trade-offs in more depth. Where an audit trail matters, write each reservation to an append-only ledger, as we did in the Solidex lead distribution engine, which also runs on caps and a full audit ledger.
What is tiered model routing, and how does it cut cost?
Tiered routing sends each request to the cheapest model that can do the job: a small, fast model for triage and extraction, and a frontier model only for hard reasoning. Most traffic in many systems is routine, so the expensive tier handles a minority of calls.
How you split depends on your workload, and we won't quote a ratio, since it varies. A workable design looks like this:
- Classify the task. Use deterministic rules where you can (document type, length, customer tier) and a small model where you cannot.
- Start at the lowest tier that your evals show is adequate for that task type.
- Escalate on a defined trigger: output fails schema validation, a confidence check fails, or a required field is missing. Don't escalate on a hunch.
- Cap escalation at one step per request. One escalation per request. If the top tier also fails, send the item to a human queue or the dead-letter queue.
- Log the tier on every request so you can see cost and quality per tier.
The prerequisite is evaluation. Without a test set per task type you cannot tell whether the small tier is good enough, and routing becomes a guess. We build evals before agents for this reason.
When does semantic caching help?
Semantic caching returns a stored answer when a new request is close in meaning to an earlier one, so the model isn't called again. It helps only when your traffic repeats, and the saving tracks how repetitive your requests are, so measure before you build.
Start simpler: an exact-match cache on a normalized prompt costs almost nothing and catches true duplicates. Add embeddings and a similarity threshold only if logs show many near-duplicates. If you do:
- Scope the cache key by tenant and by prompt version, so one customer never receives another's answer and a prompt change invalidates old entries.
- Tune the similarity threshold on real examples. Too loose returns wrong answers confidently. Too strict never hits.
- Set a time-to-live on anything that depends on changing data.
- Track hit rate and wrong-hit rate. If the hit rate is low, remove the cache, because it adds a lookup to every request.
How do circuit breakers, retry caps and a dead-letter queue work together?
A retry cap limits attempts per job. A circuit breaker stops calls to the provider after repeated failures. A dead-letter queue (DLQ) holds jobs that exhausted their retries, with the error attached. Together they turn an endless loop into a bounded, visible failure.
Write the retry policy as explicit data in one place:
const retryPolicy = {
maxAttempts: 3,
baseDelayMs: 2_000,
maxDelayMs: 60_000, // exponential backoff with jitter between these
retryOn: ['timeout', 'rate_limited', 'upstream_5xx'],
neverRetry: ['schema_invalid', 'budget_exceeded', 'content_policy'],
onExhausted: 'dead_letter',
}
These are the rules we apply.
- Never retry what can't succeed. A budget error or a validation failure will fail again and cost again.
- Back off with jitter so many workers don't retry in lockstep.
- A breaker has three states: closed (normal), open (all calls rejected for a cool-off period), half-open (let one probe through). Trip on a run of consecutive failures or an error rate over a window, and also on spend burn rate.
- Keep the payload, the last error and the attempt count on every DLQ entry. Alert on depth, fix the cause, then redrive on purpose. Idempotency keys make redrives safe, as covered in our post on concurrency, locks and idempotency.
What should you alert on?
Alert on rates of change as well as totals. A daily total tells you about yesterday. A burn rate tells you about the last ten minutes. Route the urgent alerts to a person who can act and the rest to a ticket queue.
- Spend burn rate over the last hour compared with the trailing average for the same hour. Page on a large multiple.
- Cap usage at 50, 80 and 100% for each scope. The 100% event is expected to be rare and should be investigated.
- Retry ratio, meaning retries as a share of total calls. A rise is the first sign of a loop.
- Breakers opening and DLQ depth growing.
- Tokens per request (p95). Growing context shows up here before it shows up on the bill.
- Cache hit rate and tier mix. A shift means traffic or routing changed.
Test the alerts. Run a controlled loop in staging and confirm the page arrives and the cap stops it.
What is the launch checklist for an AI feature?
Before any model-backed feature takes real traffic, confirm each of these. Together they make sure the worst night costs a known, capped amount.
- Every model call goes through one gateway function that checks the spend counter.
- Caps exist for organization, user and session, with a documented fail-open or fail-closed choice.
- Maximum output tokens, maximum steps and a per-run token budget are set on every agent.
- Context is pruned before each call, and large tool outputs are truncated or summarized.
- The retry policy lists what is and is not retried, with a hard attempt limit.
- A circuit breaker wraps the provider call, and a DLQ catches exhausted jobs.
- Routing has an eval set per task type, and the tier is logged on each request.
- Any cache is scoped by tenant and prompt version.
- Burn-rate, cap, retry and DLQ alerts are live and have been tested.
- A named person owns the on-call response, with a kill switch to disable the feature.
If you want these controls designed into a system you already run, or added before launch, our engineers work inside your team and your cloud account, and you own the code. Get in touch and we can look at where your spend is uncapped today.
