This post is for engineers and technical leads building AI agents that touch money, inventory, tickets or customer records. It is the code-level companion to two earlier posts: Anti-Vibe-Coding explains why a deterministic scaffold matters, and Building Deterministic AI Agents covers risk scoring and human review. This one shows the patterns as code you can adapt.
We practice forward deployed engineering (FDE), meaning senior engineers embedded in a client team, and these four controls turn up in almost every production agent we ship. The samples are deliberately small. Treat them as reference patterns, adapt the names, types and error handling to your stack, and test them under your own load.
What does a deterministic AI agent architecture look like?
A deterministic agent wraps one probabilistic call in plain, predictable steps: claim the work with an idempotency key, reserve budget atomically, call the model, validate the output, perform the side effect with the same key, then record the outcome. The model proposes an action and the code decides whether it happens.
Trigger (webhook, queue message, schedule)
|
v
[1] Idempotency claim (Postgres) --duplicate--> return stored result
|
v
[2] Budget reserve (Redis Lua) --exhausted--> stop, write audit row
|
v
[3] Model call (frontier or small, fast model)
|
v
[4] Schema validation (Zod) --fails twice--> human review queue
|
v
[5] Side effect (ERP, payment, email) using the same idempotency key
|
v
[6] Mark claim complete + append audit row (every step writes one)
The rest of the post takes steps 1, 2, 4 and 6 in turn. Step 3 is whatever model client you use, and step 5 is your business logic.
How do idempotency keys stop an agent repeating an action?
Insert a row keyed by a unique identifier before doing the work. The first insert wins and proceeds. A duplicate hits the unique constraint, inserts nothing, and gets the stored result back, so the side effect never runs twice. Redelivered webhooks and double-clicks become harmless.
Start with a table. The request hash lets you detect a client reusing a key with a different payload, which is a bug worth rejecting.
CREATE TABLE agent_idempotency (
idempotency_key text PRIMARY KEY,
request_hash text NOT NULL,
status text NOT NULL
CHECK (status IN ('in_progress', 'succeeded', 'failed')),
response jsonb,
created_at timestamptz NOT NULL DEFAULT now(),
completed_at timestamptz
);
Claim the key with a single statement. If the statement returns a row, you own the work. If it returns nothing, someone else already claimed it.
-- $1 = idempotency key, $2 = hash of the request payload
INSERT INTO agent_idempotency (idempotency_key, request_hash, status)
VALUES ($1, $2, 'in_progress')
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING idempotency_key;
When no row comes back, read the existing record and branch on it:
SELECT request_hash, status, response
FROM agent_idempotency
WHERE idempotency_key = $1;
- A stored request hash that differs from yours means the key is being reused for a different request, so reject the call.
- Status succeeded means you return the stored response and do nothing else.
- Status in_progress means another worker is mid-flight, so return a "try again" response.
- Status failed is a per-action decision. Some actions allow a fresh attempt and some don't.
When the work finishes, close the claim. The status guard stops a late worker from overwriting a result:
UPDATE agent_idempotency
SET status = 'succeeded', response = $2, completed_at = now()
WHERE idempotency_key = $1 AND status = 'in_progress';
Workers crash, so an in_progress row can be left behind. Reclaim stale ones after a timeout you choose, using a conditional update so only one worker wins:
UPDATE agent_idempotency
SET created_at = now()
WHERE idempotency_key = $1
AND status = 'in_progress'
AND created_at < now() - interval '5 minutes'
RETURNING idempotency_key;
One caveat matters. A reclaimed claim may follow a crash that happened after the external side effect but before the row was closed. Pass the same idempotency key to the downstream system (payment providers and many ERPs accept one), so a repeat is deduplicated there too. For a production case study of row-level locking with crowds of 50,000+ fans, see FansChance and the related post on idempotent ticket minting.
How do you enforce an AI budget or quota atomically with Redis?
Use a Lua script so Redis checks the remaining budget and decrements it in a single atomic step. Two concurrent agents can't both spend the last unit. Store the budget as an integer, such as micro-dollars or tokens, because Redis counters are integers.
-- KEYS[1] = budget key, for example budget:tenant-42:2026-11-20
-- ARGV[1] = cost to reserve, as a positive integer
-- Returns {status, remaining}
-- status 1 = reserved, 0 = insufficient budget, -1 = budget not initialised
local cost = tonumber(ARGV[1])
local current = redis.call('GET', KEYS[1])
if current == false then
return {-1, 0}
end
current = tonumber(current)
if current < cost then
return {0, current}
end
local remaining = redis.call('DECRBY', KEYS[1], cost)
return {1, remaining}
Call it from Node with ioredis. Set the daily budget once with a TTL so it resets itself, then reserve before every model call:
import Redis from 'ioredis'
const redis = new Redis(process.env.REDIS_URL as string)
const RESERVE_BUDGET = [
"local cost = tonumber(ARGV[1])",
"local current = redis.call('GET', KEYS[1])",
"if current == false then return {-1, 0} end",
"current = tonumber(current)",
"if current < cost then return {0, current} end",
"local remaining = redis.call('DECRBY', KEYS[1], cost)",
"return {1, remaining}",
].join(' ')
// SET with NX so a restart never resets a partly spent budget
export async function initBudget(key: string, units: number, ttlSeconds: number) {
await redis.set(key, units, 'EX', ttlSeconds, 'NX')
}
export async function reserveBudget(key: string, cost: number) {
const [status, remaining] = (await redis.eval(RESERVE_BUDGET, 1, key, cost)) as [number, number]
if (status === -1) throw new Error('Budget not initialised for ' + key)
return { reserved: status === 1, remaining }
}
// After the call, refund the difference between the reserve and the real cost
export async function refundBudget(key: string, unused: number) {
if (unused > 0) await redis.incrby(key, unused)
}
Reserve a worst-case cost up front, then refund the unused part once you know the real token count. If the reservation fails, don't call the model. Write an audit row and stop. This is the code behind a hard spend cap, and our post on preventing an overnight LLM token leak covers the wider cost controls. We compare this pattern with database locks in Redis Lua versus database locks.
How do you validate model output and retry only once?
Parse the output against a schema. If that fails, send the exact validation errors back to the model for one corrected attempt, and if the second attempt also fails, stop and route to human review. Nothing unvalidated should reach your database or an external system.
This TypeScript example uses Zod. Money is held in integer cents to avoid floating-point drift, and a cross-field rule checks that the lines add up to the total.
import { z } from 'zod'
const LineItem = z.object({
sku: z.string().min(2).max(64),
quantity: z.number().int().positive(),
unitPriceCents: z.number().int().positive(),
})
export const Order = z
.object({
customerId: z.string().min(1),
currency: z.enum(['USD', 'GBP', 'EUR']),
lines: z.array(LineItem).min(1).max(200),
totalCents: z.number().int().positive(),
})
.refine(
(o) => o.lines.reduce((sum, l) => sum + l.quantity * l.unitPriceCents, 0) === o.totalCents,
{ message: 'totalCents must equal the sum of quantity * unitPriceCents', path: ['totalCents'] },
)
export type Order = z.infer<typeof Order>
type CallModel = (prompt: string) => Promise<string>
export async function extractOrder(callModel: CallModel, document: string) {
const base = 'Extract the order from this document as JSON only: ' + document
let prompt = base
let lastError = ''
for (let attempt = 0; attempt < 2; attempt++) {
const raw = await callModel(prompt)
let parsed: unknown
try {
parsed = JSON.parse(raw)
} catch {
lastError = 'Output was not valid JSON'
prompt = base + ' Your previous output failed: ' + lastError + '. Return corrected JSON only.'
continue
}
const result = Order.safeParse(parsed)
if (result.success) return { ok: true as const, order: result.data }
lastError = result.error.issues.map((i) => i.path.join('.') + ': ' + i.message).join('; ')
prompt = base + ' Your previous output failed validation: ' + lastError + '. Return corrected JSON only.'
}
// Two failures: do not guess. Send to a human review queue.
return { ok: false as const, error: lastError }
}
Call the budget reservation before each iteration of that loop, since a retry costs money too. A schema only checks structure, so also verify the business facts it can't, such as whether the SKU exists and whether the total falls in an expected range. Our human-in-the-loop post covers those semantic checks and review routing.
In Python the same shape works with Pydantic: define a BaseModel with constraints, call model_validate_json on the output, catch ValidationError, and feed its error text into the single retry.
How do you build an append-only audit log for an AI agent?
Create a table the application can only insert into and read, and write one row per decision or side effect. Add triggers that reject updates, deletes and truncates, and revoke those rights from the application role. That lets you reconstruct what an agent did and why, which is what auditors and disputes ask for.
CREATE TABLE agent_audit_log (
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
occurred_at timestamptz NOT NULL DEFAULT now(),
run_id uuid NOT NULL,
idempotency_key text,
actor text NOT NULL, -- 'agent', 'human:alice', 'system'
event_type text NOT NULL, -- 'claimed', 'budget_denied', 'validated', 'executed'
payload jsonb NOT NULL
);
CREATE INDEX agent_audit_log_run_idx ON agent_audit_log (run_id, id);
CREATE FUNCTION agent_audit_log_block_changes() RETURNS trigger
LANGUAGE plpgsql AS $$
BEGIN
RAISE EXCEPTION 'agent_audit_log is append-only';
END;
$$;
CREATE TRIGGER agent_audit_log_no_update_delete
BEFORE UPDATE OR DELETE ON agent_audit_log
FOR EACH ROW EXECUTE FUNCTION agent_audit_log_block_changes();
CREATE TRIGGER agent_audit_log_no_truncate
BEFORE TRUNCATE ON agent_audit_log
FOR EACH STATEMENT EXECUTE FUNCTION agent_audit_log_block_changes();
-- The application role gets insert and select only
REVOKE ALL ON agent_audit_log FROM PUBLIC;
GRANT INSERT, SELECT ON agent_audit_log TO app_role;
Write a row at every step in the architecture diagram: the claim, any budget denial, the validated output (or the validation errors), the side effect and its result, and any human approval. Log the model's raw output and the identifiers it was given, and keep secrets and unneeded personal data out of the payload.
A table owner or superuser can still bypass these protections. For stronger tamper evidence, add a hash of the previous row to each new row, or export rows to write-once storage. The Solidex lead distribution engine keeps a full audit ledger of this kind across up to about 200,000 daily events.
What is the checklist for shipping these four controls?
Before an agent handles real volume, confirm each control exists, has been tested under concurrent load and has an owner. The list below works as a pull-request review gate.
- Every side-effecting action derives an idempotency key from the source event, and the key is passed downstream.
- The claim, complete and stale-reclaim paths each have a test, including two workers racing for the same key.
- A budget is initialized per tenant or per day, reserved before each model call, and refunded after.
- Budget exhaustion produces a logged stop and an alert.
- Model output passes schema and business-rule validation, with one retry and then a review queue.
- The audit table is append-only, the app role cannot change history, and every step writes a row.
These controls sit in our production FinOps and resilience work. If you're building an agent that has to behave predictably under load, you can contact us and we'll review the design with you.
