Exactly once belongs in the ledger, not the queue

“Exactly once” is not a useful promise about how many times a payment message travels. It is a promise about the effect that survives: one capture, one refund, one transfer, one balanced posting. The queue can reduce duplicate delivery. Only the application and its ledger can decide that two deliveries represent the same business command and ensure that command changes financial state once.

That shifts the design contract. Messages may repeat. Workers may restart. Responses may vanish. Yet a stable operation identity, an idempotent handler, an atomic ledger transaction and a transactional outbox make every replay converge on the same financial result. This is the end-to-end argument applied to money: a lower layer cannot completely implement a function that requires knowledge held by the application at the endpoints.1

The queue cannot see the effect

Consider a worker consuming capture_requested.

It calls a payment endpoint. The capture succeeds, but the response is lost. Before the worker records the result or acknowledges the message, it crashes. The visibility timeout expires and another worker receives the message.

Every component has behaved reasonably. The queue redelivered work that was not acknowledged. The worker cannot tell whether the remote effect happened. If the retry carries a new identity, it can create a second capture. If the worker acknowledges before calling the endpoint, the inverse failure can lose the capture request. Moving the acknowledgement earlier or later merely chooses between loss and duplication.

Broker documentation exposes this boundary. Kafka can atomically commit consumed offsets with output written to Kafka topics. For an external destination, however, its documentation says the consumer position must be coordinated with the stored output; the destination has to cooperate.2 Google Cloud Pub/Sub’s exactly-once guarantee is limited to pull subscriptions connected within one region, and its documentation warns that publish-side retries can still create multiple messages with different IDs.3

Those are valuable transport guarantees. They are simply not proof that a customer was charged once. A broker knows message IDs, offsets and acknowledgements. It does not know that capture-2 is a retry of capture-1, that a refund must not exceed the captured amount, or that two ledger entries must balance.

Give the business command a durable identity

The deduplication key should identify intent, not an attempt. A useful key might be capture:{payment_id}:{capture_sequence}. It stays constant across HTTP retries, queue redeliveries, worker restarts and failover. A newly generated worker UUID does not. Neither does a broker message ID if a producer can publish the same intent twice.

At the receiving boundary, store that key under a unique constraint, together with an operation type, a fingerprint of the immutable request fields, status and result reference. The first insert claims the command. A retry with the same key and fingerprint receives the existing result or an “in progress” response; the same key with different parameters is rejected. Stripe documents this pattern at its API boundary: it stores the first result for an idempotency key and rejects reuse when parameters differ.4 PayPal likewise uses a caller-supplied request ID and notes that simultaneous requests with the same ID may cause the second to fail.5

The key check and the local financial effect must share one transaction. A separate “have I seen this?” read followed by a write has a race: two workers can both observe absence. A cache-only lock also leaves correctness dependent on lock expiry and cache durability. Prefer a database uniqueness decision followed, in the same transaction, by the state transition and ledger entries.

An external payment call creates another boundary. Pass the same stable key downstream where the API supports it. After a timeout, retry or retrieve status with that identity; do not mint a fresh one. Adyen describes idempotent payment retries as stateful comparison against a retained store, and returns a transient error if that store is unavailable.6 That detail matters: idempotency is persisted state, not a clever retry policy.

Retention is part of the contract. A deduplication record must live at least as long as the system permits the command to reappear—from delayed redelivery, disaster recovery, manual replay or an upstream retry. If the record is discarded sooner, an old command becomes new again.

Use an outbox to make publication boring

After applying a command, a service often needs to publish capture_recorded. Writing the ledger and then sending to a broker is a dual write. A crash between them leaves a committed capture with no event. Sending first creates the mirror image: an event can escape even if the database transaction rolls back.

The transactional outbox removes that gap. In one database transaction, write the operation record, financial entries, aggregate state and an outbox row. A relay later publishes committed outbox rows. AWS’s description of the pattern makes both limits explicit: the database change and outbox record commit together, while the relay can still emit duplicates, so consumers must be idempotent and track processed messages.7

This is a feature, not an embarrassment. If the relay publishes and crashes before marking the row sent, it publishes again. No financial ambiguity follows because the consumer’s inbox or operation table rejects the already-applied command. Delivery remains at least once; the effect converges to once.

Ordering also becomes an application concern. Give events an aggregate ID and monotonic sequence. A consumer can reject stale transitions, defer a gap, or rebuild from the source of truth. Do not chase global ordering by default; what matters is that commands that contend for the same authorization or balance are serialized at the ledger boundary.

Put the invariant where money moves

Deduplication prevents one command from becoming two. It does not prove that a new command is valid. That is the ledger’s second job.

For each currency, debit and credit entries should balance; postings that belong together should succeed or fail atomically; a posted entry should not be edited into a different history. Public ledger documentation describes these guarantees directly: Modern Treasury specifies balanced entries, immutable posted entries, unique external transaction identifiers and atomic writes.8 TigerBeetle similarly defines equal total debits and credits, immutable transfers and balance constraints enforced when transfers are created.9

Now replay the failed capture. The first execution created operation capture:pay_42:1 and a balanced posting for 60. The second execution finds the same operation and returns its recorded outcome. It does not post again. A legitimate second partial capture uses a different sequence, but the ledger validates it against the remaining authorization. A refund is another command and another posting, constrained by captured value. Adyen’s public rules, for example, prevent total captures from exceeding the authorization and total refunds from exceeding the captured amount.6

Corrections should be new compensating entries, not mutations of old ones.89 The result is an audit trail that answers both questions separately: “How many attempts occurred?” from messages and operation records; “What money movement took effect?” from immutable postings.

Test the seam, not the happy path

The decisive tests inject failure at the uncertain moments:

• after the remote effect but before its response;
• after the ledger commit but before outbox publication;
• after publication but before the relay marks the row sent;
• during two concurrent deliveries with the same key;
• when the same key arrives with a different amount;
• when capture and refund events arrive out of order.

Each test should finish by checking ledger invariants and the durable operation record, not by counting deliveries. Duplicate-delivery rate is an operational signal. Duplicate postings, unbalanced transactions, exhausted authorizations, missing outbox events and operations stuck without a resolvable outcome are correctness signals.

Queues should still provide durability, backpressure, replay and the strongest affordable ordering or deduplication features. Those capabilities reduce work and simplify recovery. They do not carry the financial guarantee. Exactly once is achieved when every route through retries and crashes reaches one durable business outcome—and the ledger refuses every route that would create another.

Sources

1. End-to-End Arguments in System Design — J. H. Saltzer, D. P. Reed and D. D. Clark; ACM Transactions on Computer Systems / MIT. https://web.mit.edu/Saltzer/www/publications/endtoend/endtoendA4.pdf (opens in a new tab)
2. Design: Message Delivery Semantics — Apache Kafka. https://kafka.apache.org/43/design/design/#message-delivery-semantics (opens in a new tab)
3. Exactly-once delivery — Google Cloud Pub/Sub documentation. https://docs.cloud.google.com/pubsub/docs/exactly-once-delivery (opens in a new tab)
4. Idempotent requests — Stripe API Reference. https://docs.stripe.com/api/idempotent_requests (opens in a new tab)
5. Idempotency — PayPal Developer. https://developer.paypal.com/reference/guidelines/idempotency/ (opens in a new tab)
6. API idempotency — Adyen Docs. https://docs.adyen.com/development-resources/api-idempotency (opens in a new tab)
7. Transactional outbox pattern — AWS Prescriptive Guidance. https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html (opens in a new tab)
8. Ledgers Guarantees — Modern Treasury Docs. https://docs.moderntreasury.com/ledgers/docs/ledgers-guarantees (opens in a new tab)
9. TigerBeetle documentation — TigerBeetle. https://docs.tigerbeetle.com/single-page/ (opens in a new tab)