Kaprin
All insightsEngineering Briefs

AI FinOps: Preventing the Overnight LLM Token Leak

By Kaprin Engineering 10 min read

This is for CTOs and finance partners who run, or are about to run, AI features that call a paid model API. The problem is simple: model spend has no natural ceiling. A runaway loop or a retry storm can spend all night, a long conversation can do the same, and the first alert is often the bill.

Take a hypothetical, and to be clear it isn't a real incident. An agent retries in an unthrottled loop overnight. A tool it depends on starts returning errors at midnight, and the agent treats each error as a reason to try again, resending a longer history every time. Nothing crashes and no alarm fires. By morning it has made an enormous number of calls, and every one was billed.

We practice forward deployed engineering (FDE), where senior engineers work inside a client's team, and we build the controls below into production FinOps and resilience work. They're deterministic code around a non-deterministic model, the same stance as in our field guide to vibe-coded failure modes.

What stops an LLM bill from running away?

Four controls, all enforced in code: real-time spend counters with hard caps, tiered model routing, semantic caching where traffic repeats, and circuit breakers with a retry cap and dead-letter queue. A prompt asking the model to be economical is not a control.

ControlStopsWhere it runs
Spend counters and capsUnbounded total spendBefore every model call
Tiered routingUsing an expensive model for easy workRouter in front of the model
Semantic cachePaying repeatedly for the same answerBefore routing
Breaker, retry cap, DLQRetry storms and poison jobsAround every call and in the queue

How do you build real-time spend counters?

Keep one atomic counter per scope (organization, user, session) in Redis. Before each model call, reserve the estimated cost across all scopes in one step. If any scope would pass its cap, refuse the call. After the response, reconcile the reservation with actual token usage.

The check and the increment must be a single atomic operation. If you read a counter in one command and increment it in another, two concurrent calls both pass the check. A Redis Lua script runs without interruption, so nothing can interleave with it. Store cost as an integer (for example micro-dollars) because INCRBY only accepts integers.

-- KEYS[1..3]: org, user, session counters (give them one hash tag for Redis Cluster)
-- ARGV[1]: estimated cost (integer)
-- ARGV[2..4]: caps for org, user, session
-- ARGV[5..7]: TTL in seconds for org, user, session counters
local cost = tonumber(ARGV[1])
for i = 1, 3 do
  local current = tonumber(redis.call('GET', KEYS[i]) or '0')
  if current + cost > tonumber(ARGV[i + 1]) then
    return {0, i}   -- blocked; i says which scope: 1 org, 2 user, 3 session
  end
end
for i = 1, 3 do
  redis.call('INCRBY', KEYS[i], cost)
  if redis.call('TTL', KEYS[i]) < 0 then
    redis.call('EXPIRE', KEYS[i], tonumber(ARGV[i + 4]))
  end
end
return {1, 0}

Calling it from Node with ioredis looks like this:

const [allowed, scope] = await redis.eval(
  RESERVE_LUA, 3,
  '{org:42}:spend:day', '{org:42}:spend:user:7', '{org:42}:spend:session:abc',
  estimatedCost, orgCap, userCap, sessionCap, 86400, 86400, 3600
)
if (allowed !== 1) throw new SpendCapExceeded(scope)

Estimate the worst case before the call: input tokens you are about to send plus the maximum output tokens you allow. After the call, apply the difference between estimate and actual with a second INCRBY (negative if you over-reserved). Cap the maximum output length on every request, or the estimate means nothing.

Decide in advance what happens if Redis is unavailable. For agents and batch jobs we fail closed: no counter, no call. Interactive features may fall back to a small local cap. Our post on Redis Lua versus database locks covers the atomicity trade-offs in more depth. Where an audit trail matters, write each reservation to an append-only ledger, as we did in the Solidex lead distribution engine, which also runs on caps and a full audit ledger.

What is tiered model routing, and how does it cut cost?

Tiered routing sends each request to the cheapest model that can do the job: a small, fast model for triage and extraction, and a frontier model only for hard reasoning. Most traffic in many systems is routine, so the expensive tier handles a minority of calls.

How you split depends on your workload, and we won't quote a ratio, since it varies. A workable design looks like this:

  1. Classify the task. Use deterministic rules where you can (document type, length, customer tier) and a small model where you cannot.
  2. Start at the lowest tier that your evals show is adequate for that task type.
  3. Escalate on a defined trigger: output fails schema validation, a confidence check fails, or a required field is missing. Don't escalate on a hunch.
  4. Cap escalation at one step per request. One escalation per request. If the top tier also fails, send the item to a human queue or the dead-letter queue.
  5. Log the tier on every request so you can see cost and quality per tier.

The prerequisite is evaluation. Without a test set per task type you cannot tell whether the small tier is good enough, and routing becomes a guess. We build evals before agents for this reason.

When does semantic caching help?

Semantic caching returns a stored answer when a new request is close in meaning to an earlier one, so the model isn't called again. It helps only when your traffic repeats, and the saving tracks how repetitive your requests are, so measure before you build.

Start simpler: an exact-match cache on a normalized prompt costs almost nothing and catches true duplicates. Add embeddings and a similarity threshold only if logs show many near-duplicates. If you do:

  • Scope the cache key by tenant and by prompt version, so one customer never receives another's answer and a prompt change invalidates old entries.
  • Tune the similarity threshold on real examples. Too loose returns wrong answers confidently. Too strict never hits.
  • Set a time-to-live on anything that depends on changing data.
  • Track hit rate and wrong-hit rate. If the hit rate is low, remove the cache, because it adds a lookup to every request.

How do circuit breakers, retry caps and a dead-letter queue work together?

A retry cap limits attempts per job. A circuit breaker stops calls to the provider after repeated failures. A dead-letter queue (DLQ) holds jobs that exhausted their retries, with the error attached. Together they turn an endless loop into a bounded, visible failure.

Write the retry policy as explicit data in one place:

const retryPolicy = {
  maxAttempts: 3,
  baseDelayMs: 2_000,
  maxDelayMs: 60_000,   // exponential backoff with jitter between these
  retryOn: ['timeout', 'rate_limited', 'upstream_5xx'],
  neverRetry: ['schema_invalid', 'budget_exceeded', 'content_policy'],
  onExhausted: 'dead_letter',
}

These are the rules we apply.

  • Never retry what can't succeed. A budget error or a validation failure will fail again and cost again.
  • Back off with jitter so many workers don't retry in lockstep.
  • A breaker has three states: closed (normal), open (all calls rejected for a cool-off period), half-open (let one probe through). Trip on a run of consecutive failures or an error rate over a window, and also on spend burn rate.
  • Keep the payload, the last error and the attempt count on every DLQ entry. Alert on depth, fix the cause, then redrive on purpose. Idempotency keys make redrives safe, as covered in our post on concurrency, locks and idempotency.

What should you alert on?

Alert on rates of change as well as totals. A daily total tells you about yesterday. A burn rate tells you about the last ten minutes. Route the urgent alerts to a person who can act and the rest to a ticket queue.

  • Spend burn rate over the last hour compared with the trailing average for the same hour. Page on a large multiple.
  • Cap usage at 50, 80 and 100% for each scope. The 100% event is expected to be rare and should be investigated.
  • Retry ratio, meaning retries as a share of total calls. A rise is the first sign of a loop.
  • Breakers opening and DLQ depth growing.
  • Tokens per request (p95). Growing context shows up here before it shows up on the bill.
  • Cache hit rate and tier mix. A shift means traffic or routing changed.

Test the alerts. Run a controlled loop in staging and confirm the page arrives and the cap stops it.

What is the launch checklist for an AI feature?

Before any model-backed feature takes real traffic, confirm each of these. Together they make sure the worst night costs a known, capped amount.

  1. Every model call goes through one gateway function that checks the spend counter.
  2. Caps exist for organization, user and session, with a documented fail-open or fail-closed choice.
  3. Maximum output tokens, maximum steps and a per-run token budget are set on every agent.
  4. Context is pruned before each call, and large tool outputs are truncated or summarized.
  5. The retry policy lists what is and is not retried, with a hard attempt limit.
  6. A circuit breaker wraps the provider call, and a DLQ catches exhausted jobs.
  7. Routing has an eval set per task type, and the tier is logged on each request.
  8. Any cache is scoped by tenant and prompt version.
  9. Burn-rate, cap, retry and DLQ alerts are live and have been tested.
  10. A named person owns the on-call response, with a kill switch to disable the feature.

If you want these controls designed into a system you already run, or added before launch, our engineers work inside your team and your cloud account, and you own the code. Get in touch and we can look at where your spend is uncapped today.

Questions readers ask

What is AI FinOps?

AI FinOps is the practice of measuring, limiting and attributing the cost of AI usage, mainly model tokens, the way cloud FinOps does for infrastructure. It covers spend counters, budgets, model routing, caching and alerting, so cost is a controlled input.

How do you set a hard spend cap on LLM usage?

Keep an atomic counter per scope (organization, user, session) in a fast store such as Redis. Before each model call, reserve the estimated cost. If the reservation would exceed the cap, refuse the call. After the call, reconcile with actual token usage.

Does semantic caching reduce LLM costs?

It can, but the saving depends on how repetitive your traffic is. If users ask near-identical questions, hit rates can be useful. If every request is unique, a cache adds cost and complexity for little return. Measure your hit rate before committing.

What should happen when an LLM call keeps failing?

Retry a small, fixed number of times with exponential backoff and jitter, never retry errors that cannot succeed, and then move the job to a dead-letter queue. A circuit breaker should also stop calls to the provider entirely after repeated failures.

Should a spend limit fail open or closed if Redis is down?

Decide it explicitly. For autonomous agents and batch jobs, fail closed: no counter, no model call. For interactive features you may fail open with a much lower fallback cap. The worst choice is not choosing and finding out during an outage.

Bring us the broken prototype, the SaaS bill, or the manual process

Tell us what is stuck. We reply within 24 hours with a scoped plan, a fixed timeline and the one number we will move.

More insights

FDE = forward deployed engineering. Strict NDA protection. Senior architect review.