Idempotency, or How Not to Charge a Card Twice

A few years ago I got paged at 2 a.m. because a customer had been charged $240 three times for a single order. Three identical charges, three identical amounts, all within about eight seconds. The customer was, understandably, furious. Support was confused. And the logs told a story that anyone who has built payment systems will recognize instantly.

The customer's phone was on a flaky connection. They tapped "Pay". The mobile client fired a POST /charges. The request reached our server, we called the payment provider, the card was charged, the provider returned 200 OK — and then the response died somewhere on the way back to the phone. The phone saw a timeout. It had no idea whether the charge had succeeded, so it did the only reasonable thing a client can do: it retried. Twice.

That is the entire problem in one paragraph. In a distributed system you cannot tell the difference between "my request failed" and "my request succeeded but the acknowledgement got lost." A timeout is not an answer. It is the absence of an answer.

Why this is not a bug you can "just fix"

It is tempting to treat the double charge as a client bug — "the app shouldn't retry." But retries are not optional. Networks drop packets. Load balancers recycle connections. Mobile radios hand off between towers. If your client gives up after one failed attempt, you trade double charges for a worse failure: customers who paid, but whose order never went through because the success response was lost.

So every serious messaging and RPC system gives you at-least-once delivery — a message will be delivered one or more times. The alternative, at-most-once, drops messages on failure, which is unacceptable for money. At-least-once means duplicates are not an edge case. They are a guarantee. Your job is not to prevent retries; it is to make retries safe.

The tool for that is idempotency.

What idempotency actually means

A mathematician would say an operation is idempotent if applying it many times has the same effect as applying it once: f(f(x)) = f(x). For an API, the practical definition is:

Sending the same request N times produces the same result — and the same side effects — as sending it once.

HTTP already classifies its methods this way. GET, PUT, and DELETE are defined as idempotent; GET is additionally safe (no side effects at all). POST is neither — and "create a charge" is almost always a POST. That mismatch is the whole reason we have work to do. We want POST /charges to behave, under retries, like a safe method. HTTP will not give us that for free, so we build it.

A quick contrast:

GET  /balance     → safe, no side effects, retry freelyPUT  /users/42    → idempotent, last write winsPOST /charges     → NOT idempotent by default; retries duplicate

Idempotency keys

The standard solution, popularized by Stripe and now adopted across the industry, is the idempotency key: a unique, client-generated token attached to a mutating request, conventionally via an Idempotency-Key header.

POST /v1/charges HTTP/1.1Idempotency-Key: 0b8f3e2a-7c2e-4f9a-9d1e-3c5a1b2d4e6fContent-Type: application/json
{ "amount": 24000, "currency": "usd", "source": "tok_visa" }

The key is generated once per logical operation — when the user taps "Pay" — and reused across every retry of that operation. A UUIDv4 is fine. The critical rule: the client generates the key before the first attempt and holds it constant across retries. If the client mints a fresh key on each retry, you are back to square one.

A few design decisions matter here:

• Scope. A key should be unique within a meaningful namespace — typically per endpoint and per account. You do not want one tenant's key to collide with another's. I scope on (account_id, endpoint, key).
• Lifetime. Keys are not eternal. Stripe expires them after 24 hours; that is a sane default. The retry window for a single operation is seconds to minutes, so a 24-hour TTL is generous and keeps the table bounded.
• Ownership. Server-issued keys are possible but awkward — you would need a round trip to fetch one before charging. Client-generated keys avoid that. Let the client own the key.

The server algorithm

Here is the core. You need somewhere to record keys, and a relational table with a unique constraint is the workhorse — the constraint is what makes the whole thing correct under concurrency.

CREATE TABLE idempotency_keys (  id            BIGSERIAL   PRIMARY KEY,  account_id    BIGINT      NOT NULL,  endpoint      TEXT        NOT NULL,  idem_key      TEXT        NOT NULL,  request_hash  TEXT        NOT NULL,  status        TEXT        NOT NULL DEFAULT 'in_progress', -- in_progress | completed  response_code INT,  response_body JSONB,  created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),  locked_at     TIMESTAMPTZ,  UNIQUE (account_id, endpoint, idem_key));

The algorithm in prose:

1. Hash the request body.
2. Try to INSERT a new row in in_progress state. The unique constraint means exactly one concurrent request wins this insert.
3. If the insert succeeds, you are first. Do the work, store the response, mark completed.
4. If the insert fails with a conflict, a row already exists. Compare the stored request_hash to yours, then either replay the stored response (if completed) or signal "still in progress" (if not).

And the handler — pseudocode-ish TypeScript over node-postgres. Note that I keep SQL as parameterized strings, never string-interpolated, both to dodge injection and to keep the values out of the query text:

import { createHash } from 'node:crypto'import type { PoolClient } from 'pg'
function hashRequest(body: unknown): string {  // NOTE: canonicalize (sort keys) before hashing — see "Pitfalls" below.  return createHash('sha256').update(JSON.stringify(body)).digest('hex')}
export async function handleCharge(req: ChargeRequest, db: PoolClient) {  const key = req.headers['idempotency-key']  if (!key) return { code: 400, body: { error: 'Idempotency-Key required' } }
  const reqHash = hashRequest(req.body)  const endpoint = 'POST /charges'
  // Step 1 — claim the key. ON CONFLICT makes the INSERT a no-op  // if the (account_id, endpoint, idem_key) row already exists.  const claim = await db.query(    'INSERT INTO idempotency_keys (account_id, endpoint, idem_key, request_hash, status) ' +      "VALUES ($1, $2, $3, $4, 'in_progress') " +      'ON CONFLICT (account_id, endpoint, idem_key) DO NOTHING RETURNING id',    [req.accountId, endpoint, key, reqHash]  )
  // Step 2 — rowCount 0 means we lost the race, or this is a replay.  if (claim.rowCount === 0) {    const existing = await db.query(      'SELECT request_hash, status, response_code, response_body FROM idempotency_keys ' +        'WHERE account_id = $1 AND endpoint = $2 AND idem_key = $3',      [req.accountId, endpoint, key]    )    const row = existing.rows[0]
    // Same key, different body → the client is misusing the key.    if (row.request_hash !== reqHash) {      return { code: 422, body: { error: 'Idempotency-Key reused with a different body' } }    }    // Still running → tell the caller to back off and retry.    if (row.status !== 'completed') {      return { code: 409, body: { error: 'A request with this key is in progress' } }    }    // Completed → replay the stored response verbatim.    return { code: row.response_code, body: row.response_body, replayed: true }  }
  // Step 3 — we won the claim. Do the real work exactly once, and forward the  // SAME key to the provider so its layer dedupes our downstream retries too.  const charge = await paymentProvider.charge({    amount: req.body.amount,    currency: req.body.currency,    source: req.body.source,    idempotencyKey: key,  })
  // Step 4 — persist the response so future retries replay it.  await db.query(    'UPDATE idempotency_keys SET status = $1, response_code = $2, response_body = $3 ' +      'WHERE account_id = $4 AND endpoint = $5 AND idem_key = $6',    ['completed', 200, charge, req.accountId, endpoint, key]  )
  return { code: 200, body: charge }}

Concurrent duplicates: the unique constraint is the referee

The subtle part is step 2. Two retries can arrive within milliseconds — close enough that both are executing before either has finished. They both try to insert the same key. The database's unique constraint guarantees exactly one insert succeeds; the other gets a conflict. This is the single most important property of the design: you are delegating mutual exclusion to the one component that is genuinely good at it — the transactional store — instead of trying to coordinate it in application code with locks you will get subtly wrong.

The loser of the race sees in_progress and gets a 409. What should the client do with that? Back off and retry after a short delay. By then the winner has usually finished and written completed, so the next attempt replays the real response. Some teams prefer the loser to block — poll the row for a second or two before responding — so the caller gets the final answer in one shot. Both are valid; returning 409 quickly is simpler, and that is my default.

Key reuse with a different body

This one bites people. What if the same key arrives with a different payload — say, $240 the first time and $2,400 the second? That is not a retry; it is either a client bug or an attack. If you blindly replay the first response, you confirm an order the client now thinks costs ten times as much. If you process it as new, you have broken the idempotency contract.

That is why we store request_hash. On replay, compare the incoming body's hash to the stored one. If they differ, reject with 422 Unprocessable Entity (Stripe uses 400; pick one and document it). The key is a promise that "this is the same operation" — a different body breaks the promise, and the right move is to fail loudly rather than guess.

Exactly-once is a myth

I want to kill a phrase that shows up in every design doc: "exactly-once delivery." It does not exist. You cannot build it, Stripe cannot build it, Kafka cannot build it. The honest framing — and the title of a well-known essay I link below — is that you get at-least-once delivery plus idempotent processing, and the combination looks like exactly-once from the outside. The dedup happens at the consumer, not on the wire.

Once you internalize that, the same pattern shows up everywhere money moves:

• Webhook consumers. Stripe and Paystack will deliver the same charge.succeeded event more than once. Every event carries a stable id. Keep a processed_events(event_id) table, insert before you act, and let the unique constraint drop the duplicate.
• Queue consumers. SQS, Kafka, RabbitMQ — all at-least-once. An idempotent consumer dedupes on a message key exactly the way the HTTP handler dedupes on the idempotency key. Same table, same constraint, same logic.

async function onWebhook(event: ProviderEvent, db: PoolClient) {  // Insert the event id first; the unique PK makes a duplicate a no-op.  const claimed = await db.query(    'INSERT INTO processed_events (event_id) VALUES ($1) ON CONFLICT DO NOTHING RETURNING event_id',    [event.id]  )  if (claimed.rowCount === 0) return // already handled — drop the duplicate
  await applyEvent(event) // ideally in the same transaction as the insert}

There is a subtlety even here: the insert and applyEvent should run in the same transaction (or applyEvent must itself be idempotent), otherwise you can crash between them and lose the event. More on partial failures below.

Payments specifics

A few things that are particular to charging cards rather than to generic APIs:

• Providers give you idempotency keys too — use them. Notice I forwarded idempotencyKey: key to the provider in the handler. Your own idempotency layer protects you from duplicate inbound requests; the provider's key protects the leg between you and the provider. If your service crashes after calling the provider but before recording the result, your retry resends the same key downstream and the provider returns the original charge instead of creating a new one. Belt and suspenders — and both belts are load-bearing.
• Capture and refund must be idempotent too. A double capture on an authorization, or a double refund, is the same class of bug as a double charge — just with the sign flipped. A double refund costs you money. The same key discipline applies.
• Put your own layer in front of the provider. Do not rely on the provider alone. You want a record in your own database before and after every provider call, because you reconcile against your own ledger and because providers occasionally have outages and return ambiguous responses of their own.

Reconciliation: the backstop you will be glad you built

Idempotency keys make the in-band path safe. They do not make it certain. There will always be a window where you called the provider, the charge happened, and you never recorded it — process killed, database briefly unreachable, a deploy at the worst possible moment.

That is why money systems are never built on a single in-band response. They are built on reconciliation: a scheduled job that pulls the provider's record of truth — charges, refunds, payouts — and compares it line by line against your ledger. Anything in the provider's books that is missing from yours, or vice versa, gets flagged for repair.

// Scheduled hourly/daily. The provider is the source of truth for money.async function reconcile(day: string, db: PoolClient) {  const providerCharges = await paymentProvider.listCharges({ date: day })
  for (const c of providerCharges) {    const local = await db.query(      'SELECT id, amount FROM ledger_entries WHERE provider_charge_id = $1',      [c.id]    )    if (local.rowCount === 0) {      await flagDiscrepancy('missing_local', c) // charged, but we never recorded it    } else if (local.rows[0].amount !== c.amount) {      await flagDiscrepancy('amount_mismatch', c) // worse — investigate immediately    }  }}

The rule of thumb I live by: never trust a single synchronous response for anything involving money. The in-band path is the fast path; reconciliation is the truth.

Pitfalls I have actually hit

• Storing keys forever. The table grows without bound and your INSERT latency creeps up. Set a TTL — match the provider's 24h — and run a periodic DELETE FROM idempotency_keys WHERE created_at < now() - interval '24 hours'. Bounded tables stay fast.
• Non-deterministic responses. If you store and replay a response, the replay must equal what the first call returned. If your response embeds a fresh generated_at timestamp or a new server-side id on each call, store the original response and replay it verbatim — do not regenerate it. The whole point is that retries see an identical answer.
• Non-canonical hashing. JSON.stringify is order-sensitive: {a:1,b:2} and {b:2,a:1} hash differently. Canonicalize (sort keys) before hashing, or you will reject legitimate retries as a "different body."
• Side effects outside your database. The classic: you mark the charge complete, then send a receipt email — and crash before recording that the email went out. The retry sends a second email. Emails, SMS, push, downstream API calls — none of them participate in your transaction. Push them onto an idempotent queue (dedup on a key) instead of firing them inline, so "send receipt" itself becomes a once-per-key operation.
• Partial failures and the in_progress tombstone. If the winner crashes after claiming the key but before writing completed, you are left with a stranded in_progress row, and every retry now gets a 409 forever. Use locked_at: if a row has been in_progress longer than some timeout, treat it as abandoned and let a retry take it over — carefully, ideally after asking the provider (via its idempotency key) whether the original work actually landed.

Wrapping up

Idempotency is not a feature you bolt on after the double-charge incident — it is the shape of any system that touches money. The mental model is small, and it composes:

1. Retries are guaranteed, not exceptional — design for at-least-once.
2. Give every mutating operation a client-generated key.
3. Let a unique constraint, not application locks, pick the one winner.
4. Store the response and replay it; reject the same key with a different body.
5. Back all of it with reconciliation, because the in-band response can always be lost.

Do those five things and "charged twice" stops being a 2 a.m. page and becomes a duplicate request your system quietly absorbs — which is exactly what your customers should never have to think about.

Further reading

• Stripe — Idempotent requests: the canonical reference for the Idempotency-Key header and 24-hour expiry behavior.
• Paystack — API documentation on idempotency and webhook event delivery.
• Mathias Verraes — "You Cannot Have Exactly-Once Delivery": worth re-reading every time someone proposes building it.