The most expensive defect we find when reviewing payment systems is also the easiest to prevent. A request times out, the client retries, and the customer is charged twice — even though the first request succeeded, silently, after the client gave up waiting.
A timeout is not a failure
When a request times out, the client knows only that it did not receive a response. The operation may have completed perfectly. This ambiguity is unavoidable in distributed systems, which means the retry is correct behaviour — and the server must therefore be able to recognise it.
A timeout tells you nothing about whether the work happened. Design as if it did.
How the key works
The client generates a unique key per logical operation and sends it with the request. The server stores the key with the result. If the same key arrives again, it returns the stored result rather than performing the work a second time. The client must reuse the same key on retry — generating a fresh one defeats the entire mechanism.
Store the response, not just the key
Recording that a key was seen is insufficient; the retry needs the same answer the original produced, including the payment identifier. Store the full response body and status against the key, with a retention window comfortably longer than any client's retry policy — 24 hours is a common and sensible default.
The concurrent case people miss
Two identical requests can arrive before the first completes — a double-click produces exactly this. Checking whether the key exists and then processing is a race condition. Insert the key first with a unique constraint so the second request fails to claim it and waits or returns a conflict, rather than both proceeding.
It applies well beyond payments
Anything that creates a record, sends a message, triggers a workflow or calls a partner API benefits. Queue consumers especially — most queues guarantee at-least-once delivery, which means redelivery is normal operation and the consumer must tolerate it.
Test the duplicate path
Write the test that fires the same request twice concurrently and asserts one effect. It takes ten minutes and it is the only way to know the protection works. Most idempotency bugs we find are in code that has an idempotency key parameter which nothing actually enforces.
