Yazan.
ENعDE
← All articles
10 min readFintechBackendPayments

Idempotency Keys — The Unsung Hero of Every Payment System

Idempotency Keys — The Unsung Hero of Every Payment System

Think of an idempotency key like a receipt number.

You submit a transaction. Something fails mid-way — a timeout, a network glitch, a server hiccup. You retry. But the system already saw that receipt number. So instead of processing again, it just returns: "Already done."

No double charges. No duplicate transfers. No angry users.

That's the whole idea in one paragraph. The rest of this post is about the part that actually bites you: the version everyone writes first is wrong, and it fails exactly when you can least afford it.

Why this matters

In distributed systems, failures are not exceptional — they're routine. Networks drop packets, servers restart, mobile clients lose signal in an elevator and retry when they come out. If your payment endpoint processes every request it receives, one flaky connection charges a customer twice.

And note where the failure happens. The dangerous case is not "the request never arrived." That one is harmless — nothing happened, the retry is correct. The dangerous case is:

  1. Request arrives.
  2. Server debits the sender, credits the receiver, commits.
  3. Response is lost on the way back.
  4. Client sees a timeout and retries, because from the client's point of view nothing happened.

The client cannot tell case 1 from case 4. It has no way to know. So the client will always retry, and the server has to be the one that remembers.

The contract

  1. The client generates a unique key for each logical operation — not each HTTP request. One transfer, one key, no matter how many times it is sent.
  2. The server stores the key together with the operation's result.
  3. If the same key arrives again, the server returns the stored result instead of re-executing.

Point 1 is the one clients get wrong. If the SDK generates a fresh UUID inside the retry loop, every retry looks like a new transfer and the whole mechanism does nothing. The key belongs to the user's intent — "send 200 USD to Ahmad" — so it must be created before the first attempt and reused for every retry of that same intent.

The schema

CREATE TABLE idempotency_key (
    id              BIGSERIAL PRIMARY KEY,
    key             TEXT        NOT NULL,
    user_id         BIGINT      NOT NULL REFERENCES accounts_user(id),
    endpoint        TEXT        NOT NULL,
    request_hash    TEXT        NOT NULL,
    state           TEXT        NOT NULL DEFAULT 'in_progress',
    response_code   INTEGER,
    response_body   JSONB,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),

    CONSTRAINT idempotency_key_scope UNIQUE (user_id, endpoint, key)
);

Four decisions worth defending:

The unique constraint is on (user_id, endpoint, key), not on key alone. Keys are client-generated. Two clients can pick the same UUID — unlikely, but "unlikely" is not "impossible," and the failure mode is user A getting user B's transfer receipt. That is a data leak, not a bug. Scoping the key to the user makes the collision harmless. Adding the endpoint means a key reused across two different operations doesn't silently return the wrong one.

request_hash exists so key reuse is detectable. A client that sends the same key with a different body is buggy — it might be sending 200 USD the first time and 2,000 the second. You do not want to return the cached 200 receipt and quietly drop the second request. Hash the canonical body, compare on replay, and reject with 422 if it differs.

state distinguishes "done" from "in flight." This is the field the naive implementation leaves out, and it's the one this whole post is about.

response_body is JSONB, not a foreign key to the transaction. You replay the exact original response, including the status code. If you rebuild the response from current data, a replay three hours later can return a different body than the original — the transfer may have settled or been reversed since. Replays must be a recording, not a re-render.

The race condition

Here is the version almost everyone writes first:

# Broken. Do not ship this.
def transfer(request):
    key = request.headers["Idempotency-Key"]

    existing = IdempotencyKey.objects.filter(key=key, user=request.user).first()
    if existing:
        return Response(existing.response_body, status=existing.response_code)

    result = perform_transfer(request.data)          # money moves here

    IdempotencyKey.objects.create(
        key=key, user=request.user,
        response_body=result, response_code=201,
    )
    return Response(result, status=201)

Read it as two threads instead of one.

A mobile client on a bad connection fires the request, times out at 10 seconds, and retries. The first request is still running — it was slow, not lost. Now two requests with the same key are inside this function at the same time:

Thread A Thread B
t0 filter(key=...) → nothing
t1 filter(key=...) → nothing
t2 perform_transfer() → money moves
t3 perform_transfer() → money moves again
t4 create(...)
t5 create(...) → IntegrityError

Both threads read before either wrote. The check-then-act is not atomic, so the guard is a suggestion. The customer is debited twice, and the only sign anything went wrong is an IntegrityError in the logs after the second transfer already committed.

This is not a rare interleaving. It is the expected interleaving whenever a slow request gets retried, which is precisely the scenario idempotency exists to handle. The window is as wide as your endpoint is slow — and a payment endpoint calling an upstream provider is slow.

The fix: let the database do the checking

The insert itself has to be the lock. Write the key first, in in_progress, and let the unique constraint reject the second thread before any money moves.

from django.db import IntegrityError, transaction
from rest_framework.exceptions import ValidationError
from rest_framework.response import Response

def transfer(request):
    key = request.headers.get("Idempotency-Key")
    if not key:
        raise ValidationError({"Idempotency-Key": "required"})

    body_hash = canonical_hash(request.data)

    try:
        # The unique constraint — not the SELECT above it — is what makes this
        # safe. Exactly one concurrent request can win this INSERT.
        record = IdempotencyKey.objects.create(
            key=key,
            user=request.user,
            endpoint="transfer",
            request_hash=body_hash,
            state="in_progress",
        )
    except IntegrityError:
        return replay(key, request.user, body_hash)

    with transaction.atomic():
        result = perform_transfer(request.data)
        record.state = "completed"
        record.response_code = 201
        record.response_body = result
        record.save(update_fields=["state", "response_code", "response_body"])

    return Response(result, status=201)

And the replay path, which has three cases rather than one:

def replay(key, user, body_hash):
    record = IdempotencyKey.objects.get(user=user, endpoint="transfer", key=key)

    if record.request_hash != body_hash:
        # Same key, different payload: the client is buggy. Refusing is the
        # only safe answer — returning the cached receipt would silently
        # swallow a transfer the user believes they made.
        raise ValidationError({"Idempotency-Key": "reused with a different body"})

    if record.state == "in_progress":
        # The original is still running. Tell the client to wait rather than
        # inventing an answer; 409 is retryable, a fabricated 201 is not.
        return Response({"detail": "request in progress"}, status=409)

    return Response(record.response_body, status=record.response_code)

The 409 is not a cop-out. The alternative is blocking on the row until the first thread finishes, which converts a retry storm into a pile of held connections and takes the database down instead. Returning quickly and letting the client back off is the cheaper failure.

Details that decide whether this survives production

Keep the key row and the money in one transaction. If perform_transfer commits and the save() on the key row fails, you have moved money with no record that you did — and the next retry will move it again. Both writes belong to the same transaction.atomic() block, in the same database. An idempotency table in Redis with the ledger in Postgres cannot give you that, and this is one of the few places where the convenience is not worth it.

Crashed in_progress rows need a sweeper. If the process dies between the INSERT and the update, the key is stuck in in_progress forever, and every retry gets a 409 — the transfer becomes permanently unmakeable with that key. A periodic job should reconcile rows older than the endpoint's timeout against the ledger and resolve them, either to completed with the real result or to failed so the key can be retried.

Give keys a TTL, sized to your retry window. Twenty-four hours covers essentially every legitimate client retry. Keeping them forever turns the table into your largest and least useful index. Deleting them too early reopens the double-charge window.

Idempotency does not mean "safe to replay forever." After the TTL expires, the same key is a new operation. That is fine — no honest client retries a transfer a week later — but it should be a deliberate choice, documented for whoever integrates with you.

In fintech, retry logic without idempotency isn't safety — it's a liability. And idempotency without the unique constraint doing the work is not idempotency. It's a race with better paperwork.