The checkout button was clicked only once from the user's point of view. The browser paused, the spinner kept turning, and after a few seconds the user clicked again because nothing seemed to happen. Somewhere behind that simple moment, the payment service received two requests, the order service retried after a timeout, and support later had to explain why one person saw two pending charges for one purchase.
This is where idempotency stops being an architecture word and becomes a product promise. An idempotent operation can be safely repeated and still leave the system in the same intended state. If a client retries a request because the network was slow, the server should not accidentally create a second order, send a second email, or charge a card twice. The repeated request should either return the same result or report the current state without creating new damage.
Retries are not rare edge cases. They are part of normal distributed systems. Mobile networks drop. Browsers refresh. Load balancers time out. Message queues redeliver. A worker crashes after doing the side effect but before acknowledging the job. A user double-clicks. A client library retries automatically. If the API assumes every request is unique just because the caller meant well, the system is depending on a world that does not exist.
The simplest example is a read operation. Calling GET /orders/123 many times should not change the order. Most people understand that as safe. The harder work begins with operations that create or change state. Creating an order, starting a refund, scheduling a transfer, publishing a message, or sending a notification may have consequences outside the database. These operations need a deliberate answer to the question: what should happen if this exact intention arrives twice?
One common tool is an idempotency key. The client sends a stable key with the request, often generated when the user begins the action. The server stores that key together with the request fingerprint and the final response. If the same key arrives again, the server does not repeat the business action. It returns the saved result or a clear in-progress response. The key turns a noisy transport problem into a recognizable business attempt.
The key is not enough by itself. The server must decide what sameness means. If the same idempotency key is reused with a different amount, different currency, or different order id, that should not be quietly accepted. It should be rejected as a conflict. Otherwise the key becomes a vague label instead of a reliable contract. Good idempotency protects both sides: it prevents duplicate execution, and it prevents accidental reuse for a different intention.
State design matters too. Some actions are naturally easier to make idempotent when they are modeled as transitions instead of commands that blindly do work. Marking an invoice as paid is calmer than creating another payment record every time a webhook arrives. Recording that a refund has already been requested is calmer than calling the payment provider again. A state machine with clear terminal states often makes duplicate handling visible instead of hiding it in scattered conditionals.
External side effects need special care. If the database commit succeeds but the email send fails, should a retry create another database row or only retry the email? If the payment provider succeeds but the local service times out before saving the result, how will the system reconcile? These questions are why patterns like transactional outbox, provider-side idempotency keys, and reconciliation jobs matter. Idempotency is rarely one line of code. It is a small chain of agreements across boundaries.
Responses should also be boring and predictable. A repeated request should not surprise the caller with a completely different shape. It may return the original success response, a current resource representation, or a clear status such as already processed. What matters is that clients can build simple retry behavior without needing to guess whether a second response means a second action happened.
There is a cost. Idempotency records need storage, retention rules, cleanup, and sometimes locking. Long retention protects slow retries but stores more data. Short retention is cheaper but can make late duplicates dangerous. High-throughput APIs need to think about contention. Security-sensitive APIs need to avoid leaking whether a key exists for another user. These details are not glamorous, but they are where reliability becomes real.
I like to think of idempotent APIs as a form of patience in system design. They accept that people, networks, workers, and vendors will repeat themselves. Instead of treating repetition as bad behavior, they make repetition safe. If your product has any operation where doing it twice would be embarrassing, expensive, or hard to explain, that operation deserves an idempotency conversation before the next retry teaches the lesson in production.