Kaprin
All insightsEngineering Briefs

Deterministic AI Agents: Idempotency Keys and Atomic Locks

By Kaprin Engineering 10 min read

This post is for engineers and technical leads building AI agents that touch money, inventory, tickets or customer records. It is the code-level companion to two earlier posts: Anti-Vibe-Coding explains why a deterministic scaffold matters, and Building Deterministic AI Agents covers risk scoring and human review. This one shows the patterns as code you can adapt.

We practice forward deployed engineering (FDE), meaning senior engineers embedded in a client team, and these four controls turn up in almost every production agent we ship. The samples are deliberately small. Treat them as reference patterns, adapt the names, types and error handling to your stack, and test them under your own load.

What does a deterministic AI agent architecture look like?

A deterministic agent wraps one probabilistic call in plain, predictable steps: claim the work with an idempotency key, reserve budget atomically, call the model, validate the output, perform the side effect with the same key, then record the outcome. The model proposes an action and the code decides whether it happens.

Trigger (webhook, queue message, schedule)
   |
   v
[1] Idempotency claim (Postgres)  --duplicate-->  return stored result
   |
   v
[2] Budget reserve (Redis Lua)    --exhausted-->  stop, write audit row
   |
   v
[3] Model call (frontier or small, fast model)
   |
   v
[4] Schema validation (Zod)       --fails twice-->  human review queue
   |
   v
[5] Side effect (ERP, payment, email) using the same idempotency key
   |
   v
[6] Mark claim complete + append audit row (every step writes one)

The rest of the post takes steps 1, 2, 4 and 6 in turn. Step 3 is whatever model client you use, and step 5 is your business logic.

How do idempotency keys stop an agent repeating an action?

Insert a row keyed by a unique identifier before doing the work. The first insert wins and proceeds. A duplicate hits the unique constraint, inserts nothing, and gets the stored result back, so the side effect never runs twice. Redelivered webhooks and double-clicks become harmless.

Start with a table. The request hash lets you detect a client reusing a key with a different payload, which is a bug worth rejecting.

CREATE TABLE agent_idempotency (
  idempotency_key text PRIMARY KEY,
  request_hash    text NOT NULL,
  status          text NOT NULL
                  CHECK (status IN ('in_progress', 'succeeded', 'failed')),
  response        jsonb,
  created_at      timestamptz NOT NULL DEFAULT now(),
  completed_at    timestamptz
);

Claim the key with a single statement. If the statement returns a row, you own the work. If it returns nothing, someone else already claimed it.

-- $1 = idempotency key, $2 = hash of the request payload
INSERT INTO agent_idempotency (idempotency_key, request_hash, status)
VALUES ($1, $2, 'in_progress')
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING idempotency_key;

When no row comes back, read the existing record and branch on it:

SELECT request_hash, status, response
FROM agent_idempotency
WHERE idempotency_key = $1;
  • A stored request hash that differs from yours means the key is being reused for a different request, so reject the call.
  • Status succeeded means you return the stored response and do nothing else.
  • Status in_progress means another worker is mid-flight, so return a "try again" response.
  • Status failed is a per-action decision. Some actions allow a fresh attempt and some don't.

When the work finishes, close the claim. The status guard stops a late worker from overwriting a result:

UPDATE agent_idempotency
SET status = 'succeeded', response = $2, completed_at = now()
WHERE idempotency_key = $1 AND status = 'in_progress';

Workers crash, so an in_progress row can be left behind. Reclaim stale ones after a timeout you choose, using a conditional update so only one worker wins:

UPDATE agent_idempotency
SET created_at = now()
WHERE idempotency_key = $1
  AND status = 'in_progress'
  AND created_at < now() - interval '5 minutes'
RETURNING idempotency_key;

One caveat matters. A reclaimed claim may follow a crash that happened after the external side effect but before the row was closed. Pass the same idempotency key to the downstream system (payment providers and many ERPs accept one), so a repeat is deduplicated there too. For a production case study of row-level locking with crowds of 50,000+ fans, see FansChance and the related post on idempotent ticket minting.

How do you enforce an AI budget or quota atomically with Redis?

Use a Lua script so Redis checks the remaining budget and decrements it in a single atomic step. Two concurrent agents can't both spend the last unit. Store the budget as an integer, such as micro-dollars or tokens, because Redis counters are integers.

-- KEYS[1] = budget key, for example budget:tenant-42:2026-11-20
-- ARGV[1] = cost to reserve, as a positive integer
-- Returns {status, remaining}
--   status  1 = reserved, 0 = insufficient budget, -1 = budget not initialised
local cost = tonumber(ARGV[1])
local current = redis.call('GET', KEYS[1])
if current == false then
  return {-1, 0}
end
current = tonumber(current)
if current < cost then
  return {0, current}
end
local remaining = redis.call('DECRBY', KEYS[1], cost)
return {1, remaining}

Call it from Node with ioredis. Set the daily budget once with a TTL so it resets itself, then reserve before every model call:

import Redis from 'ioredis'

const redis = new Redis(process.env.REDIS_URL as string)

const RESERVE_BUDGET = [
  "local cost = tonumber(ARGV[1])",
  "local current = redis.call('GET', KEYS[1])",
  "if current == false then return {-1, 0} end",
  "current = tonumber(current)",
  "if current < cost then return {0, current} end",
  "local remaining = redis.call('DECRBY', KEYS[1], cost)",
  "return {1, remaining}",
].join(' ')

// SET with NX so a restart never resets a partly spent budget
export async function initBudget(key: string, units: number, ttlSeconds: number) {
  await redis.set(key, units, 'EX', ttlSeconds, 'NX')
}

export async function reserveBudget(key: string, cost: number) {
  const [status, remaining] = (await redis.eval(RESERVE_BUDGET, 1, key, cost)) as [number, number]
  if (status === -1) throw new Error('Budget not initialised for ' + key)
  return { reserved: status === 1, remaining }
}

// After the call, refund the difference between the reserve and the real cost
export async function refundBudget(key: string, unused: number) {
  if (unused > 0) await redis.incrby(key, unused)
}

Reserve a worst-case cost up front, then refund the unused part once you know the real token count. If the reservation fails, don't call the model. Write an audit row and stop. This is the code behind a hard spend cap, and our post on preventing an overnight LLM token leak covers the wider cost controls. We compare this pattern with database locks in Redis Lua versus database locks.

How do you validate model output and retry only once?

Parse the output against a schema. If that fails, send the exact validation errors back to the model for one corrected attempt, and if the second attempt also fails, stop and route to human review. Nothing unvalidated should reach your database or an external system.

This TypeScript example uses Zod. Money is held in integer cents to avoid floating-point drift, and a cross-field rule checks that the lines add up to the total.

import { z } from 'zod'

const LineItem = z.object({
  sku: z.string().min(2).max(64),
  quantity: z.number().int().positive(),
  unitPriceCents: z.number().int().positive(),
})

export const Order = z
  .object({
    customerId: z.string().min(1),
    currency: z.enum(['USD', 'GBP', 'EUR']),
    lines: z.array(LineItem).min(1).max(200),
    totalCents: z.number().int().positive(),
  })
  .refine(
    (o) => o.lines.reduce((sum, l) => sum + l.quantity * l.unitPriceCents, 0) === o.totalCents,
    { message: 'totalCents must equal the sum of quantity * unitPriceCents', path: ['totalCents'] },
  )

export type Order = z.infer<typeof Order>

type CallModel = (prompt: string) => Promise<string>

export async function extractOrder(callModel: CallModel, document: string) {
  const base = 'Extract the order from this document as JSON only: ' + document
  let prompt = base
  let lastError = ''

  for (let attempt = 0; attempt < 2; attempt++) {
    const raw = await callModel(prompt)

    let parsed: unknown
    try {
      parsed = JSON.parse(raw)
    } catch {
      lastError = 'Output was not valid JSON'
      prompt = base + ' Your previous output failed: ' + lastError + '. Return corrected JSON only.'
      continue
    }

    const result = Order.safeParse(parsed)
    if (result.success) return { ok: true as const, order: result.data }

    lastError = result.error.issues.map((i) => i.path.join('.') + ': ' + i.message).join('; ')
    prompt = base + ' Your previous output failed validation: ' + lastError + '. Return corrected JSON only.'
  }

  // Two failures: do not guess. Send to a human review queue.
  return { ok: false as const, error: lastError }
}

Call the budget reservation before each iteration of that loop, since a retry costs money too. A schema only checks structure, so also verify the business facts it can't, such as whether the SKU exists and whether the total falls in an expected range. Our human-in-the-loop post covers those semantic checks and review routing.

In Python the same shape works with Pydantic: define a BaseModel with constraints, call model_validate_json on the output, catch ValidationError, and feed its error text into the single retry.

How do you build an append-only audit log for an AI agent?

Create a table the application can only insert into and read, and write one row per decision or side effect. Add triggers that reject updates, deletes and truncates, and revoke those rights from the application role. That lets you reconstruct what an agent did and why, which is what auditors and disputes ask for.

CREATE TABLE agent_audit_log (
  id              bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  occurred_at     timestamptz NOT NULL DEFAULT now(),
  run_id          uuid NOT NULL,
  idempotency_key text,
  actor           text NOT NULL,   -- 'agent', 'human:alice', 'system'
  event_type      text NOT NULL,   -- 'claimed', 'budget_denied', 'validated', 'executed'
  payload         jsonb NOT NULL
);

CREATE INDEX agent_audit_log_run_idx ON agent_audit_log (run_id, id);

CREATE FUNCTION agent_audit_log_block_changes() RETURNS trigger
LANGUAGE plpgsql AS $$
BEGIN
  RAISE EXCEPTION 'agent_audit_log is append-only';
END;
$$;

CREATE TRIGGER agent_audit_log_no_update_delete
  BEFORE UPDATE OR DELETE ON agent_audit_log
  FOR EACH ROW EXECUTE FUNCTION agent_audit_log_block_changes();

CREATE TRIGGER agent_audit_log_no_truncate
  BEFORE TRUNCATE ON agent_audit_log
  FOR EACH STATEMENT EXECUTE FUNCTION agent_audit_log_block_changes();

-- The application role gets insert and select only
REVOKE ALL ON agent_audit_log FROM PUBLIC;
GRANT INSERT, SELECT ON agent_audit_log TO app_role;

Write a row at every step in the architecture diagram: the claim, any budget denial, the validated output (or the validation errors), the side effect and its result, and any human approval. Log the model's raw output and the identifiers it was given, and keep secrets and unneeded personal data out of the payload.

A table owner or superuser can still bypass these protections. For stronger tamper evidence, add a hash of the previous row to each new row, or export rows to write-once storage. The Solidex lead distribution engine keeps a full audit ledger of this kind across up to about 200,000 daily events.

What is the checklist for shipping these four controls?

Before an agent handles real volume, confirm each control exists, has been tested under concurrent load and has an owner. The list below works as a pull-request review gate.

  1. Every side-effecting action derives an idempotency key from the source event, and the key is passed downstream.
  2. The claim, complete and stale-reclaim paths each have a test, including two workers racing for the same key.
  3. A budget is initialized per tenant or per day, reserved before each model call, and refunded after.
  4. Budget exhaustion produces a logged stop and an alert.
  5. Model output passes schema and business-rule validation, with one retry and then a review queue.
  6. The audit table is append-only, the app role cannot change history, and every step writes a row.

These controls sit in our production FinOps and resilience work. If you're building an agent that has to behave predictably under load, you can contact us and we'll review the design with you.

Questions readers ask

What is an idempotency key in an AI agent?

It is a unique identifier for one logical action, such as one incoming webhook or one order, stored before the action runs. If the same key arrives again, the system returns the stored result and skips the side effect. That makes retries safe.

Why use a Redis Lua script instead of separate GET and DECR calls?

Redis runs a Lua script atomically, so the check and the decrement happen as one step with no other command in between. Separate GET and DECR calls leave a gap where two concurrent agents can both see enough budget and both spend it.

Should I use Zod or Pydantic to validate model output?

Use whichever matches your stack: Zod for TypeScript, Pydantic for Python. Both turn free-form model output into typed data or a precise error message. What matters is that nothing reaches a database or external system until validation passes.

How many times should an agent retry after a validation failure?

Once is a sensible default. Feed the validation error back to the model for one corrected attempt, then route to a human review queue if it fails again. More retries mostly add cost and latency, and a model that fails twice on the same input will often fail a third time.

Does an append-only audit log need a special database?

A normal Postgres table works if you give the application role insert and select rights only, and add triggers that reject updates, deletes and truncates. For stronger tamper evidence, add hash chaining or ship rows to write-once storage.

Bring us the broken prototype, the SaaS bill, or the manual process

Tell us what is stuck. We reply within 24 hours with a scoped plan, a fixed timeline and the one number we will move.

More insights

FDE = forward deployed engineering. Strict NDA protection. Senior architect review.