# Idempotency is two mechanisms and an honest "unknown"

Payment providers retry webhooks. Guardian has to fulfil each payment exactly once anyway. This is how it does that, and the one case where it refuses to guess.

- **sheet**: 4.2
- **published**: 2026-07-12
- **reading**: 7 min
- **tags**: payments, postgres, redis

## The contract

A provider may deliver the same event many times, concurrently, and after any amount of delay. Guardian's side of the deal is that the fulfilment, the external action the payment pays for, happens once. Not at least once, not at most once. Once.

No single mechanism gives you that. Guardian uses two for deduplication, one for concurrency, and a deliberate refusal for the case the others cannot cover.

## Fast path: Redis `SET NX`

The first thing the API does with an event is try to claim a key for it.

```
SET idem:{event_id} {worker_id} NX EX 86400
```

If the key already exists, the event is a duplicate and the request ends there, cheaply, without touching the database. Duplicates tend to arrive close together, and this catches them.

It is not enough on its own. The key expires. Redis can restart and lose it. So Redis is a filter, not a record.

## Source of truth: a unique constraint

The record lives in Postgres. The event ID has a unique constraint, and the insert says what to do on conflict.

```
insert into fulfilment (event_id, payment_id, state)
values ($1, $2, 'pending')
on conflict (event_id) do nothing
returning id;
```

If a row comes back, this worker owns the event. If nothing comes back, someone else got there first. The database decides, atomically, and it does not forget.

> further reading Brandur Leach's post on Stripe-style idempotency keys in Postgres covers the same ground for API requests.

## Moving through states safely

Owning the row is not the end of it. A dispatcher still has to pick up the pending work, and two dispatchers must not both pick up the same row. Every transition is a conditional update that names the state it expects.

```
update fulfilment
   set state = 'dispatching'
 where id = $1 and state = 'pending'
returning id;
```

The loser of a race gets zero rows back and moves on. On top of that, a partial unique index guarantees that a payment can have at most one fulfilment in an active state, which catches bugs in the code as well as races.

## The honest "unknown"

Here is the case none of the above can solve. The dispatcher sends a request to the connector. The process dies before the response is recorded. On restart, the row says `dispatching`.

Did the external action happen? Maybe. The request may have been lost, or it may have succeeded a millisecond before the crash. Retrying risks fulfilling twice. Marking it failed risks never fulfilling at all.

Guardian does neither. It moves the row to `needs_reconciliation` and stops. Someone, or later a job that can ask the external system what it actually did, resolves it with facts instead of a guess.

```
pending --> dispatching --+--> fulfilled              (call returned)
                          |
                          +--> needs_reconciliation   (crashed mid-call)
```

The only two ways out of dispatching.

This is the part I would defend hardest in a review. A small queue of cases a human looks at is a better failure mode than a rare double charge that nobody notices.

## Single-use tokens

Some actions carry a token that may be used once. Reading it and deleting it has to be one step, so it runs as a Lua script inside Redis.

```
local v = redis.call('GET', KEYS[1])
if v then redis.call('DEL', KEYS[1]) end
return v
```

Redis 6.2 added `GETDEL`, which does the same thing as a single command.

## What is still wrong

The loop lock is released unconditionally. If its TTL expires and another worker takes it, the first worker's release deletes the second worker's lock. The fix is the same compare-then-delete pattern as the tokens. That one is open, and it will get its own post when the fix lands.

[Guardian case note →](https://0xshriram.dev/projects/guardian.md)

[← All posts](https://0xshriram.dev/blog.md)

---
Canonical: https://0xshriram.dev/blog/idempotency-two-mechanisms.html