Your webhook signature solves only the first attack
A provider can exhaust its retry window while the endpoint keeps returning success for broken processing. Without a durable inbox and backfill path, valid events disappear silently.
7 min read
A valid webhook signature answers one narrow question: did someone holding the signing key authenticate this signed content? HMAC verification can establish origin and integrity for the content covered by the signature; it does not establish that the message is new, unique, correctly ordered, safe for the rest of the system, or recoverable after a failure.12
That distinction matters because forgery is only the first attack. A captured valid request can be replayed.23 A legitimate provider retry can repeat a side effect.3 A later state transition can arrive before an earlier one.34 A genuine payload can still reach a vulnerable parser or an over-privileged worker.6 And a flawless verifier cannot reconstruct the events missed during an outage.10
Verify bytes, then verify time
Signature verification has to operate on the representation the sender signed. Stripe, for example, requires the raw request body and warns that any manipulation of it causes verification to fail. Its manual procedure signs the timestamp plus the JSON payload with HMAC-SHA256 and recommends a constant-time comparison.3 Parsing and reserialising JSON first is therefore not a harmless refactor: whitespace, escaping, field order, or character encoding may change the byte sequence.
Even correct byte verification leaves replay open. An attacker who records a signed request can send the same body and signature again. HTTP Message Signatures addresses this class of problem with creation and expiration times or a per-message nonce that the verifier can remember; Stripe signs a timestamp and recommends rejecting requests outside a tolerance window.23
The receiver consequently needs two checks: cryptographic validity and freshness. Reject timestamps too far in the past or future, monitor clock synchronisation, and make the accepted window as short as operational latency permits. Record the reason for rejection without putting secrets or complete payment payloads into logs.
Freshness is still not idempotency. Stripe creates a new timestamp and signature for each retry, so a legitimate second delivery can be both fresh and authentic.3 The replay cache answers “have I seen this signed request?” The idempotency ledger answers “have I already applied this business event?” They are different records with different retention periods.
Make duplicate delivery boring
Assume at-least-once delivery. Stripe says an endpoint may receive the same event more than once and recommends recording processed event IDs; it also notes that separate event objects can describe the same underlying object and event type.3 Adyen likewise documents duplicate events and identifies them using fields including eventCode and pspReference.4
The durable inbox is the useful pattern. In one database transaction, insert a row keyed by provider, receiving account or endpoint, and event ID; put a unique constraint on that key; store the minimum payload needed for processing; then acknowledge receipt. If the insert conflicts, return success and do nothing. The database constraint, not a check-then-write in application code, closes the race between concurrent deliveries.
Business effects need their own idempotency boundary. A unique event ID prevents the same envelope from being applied twice, but it cannot recognise two distinct envelopes that request the same irreversible action. Key such effects to the domain operation—payment reference plus transition, for example—and make the state change and its outbox record atomic. “Exactly once” is then a property of the local transaction, not a promise borrowed from HTTP.
Treat rotation as a protocol
A signing secret should not be a single configuration value scattered across instances. It needs an identity, a lifecycle state, an activation time, an expiry time, and an owner. Adyen recommends a different HMAC key for each endpoint and a new key when moving from test to live.1 That separation limits accidental cross-environment or cross-endpoint acceptance.
Planned rotation needs a bounded overlap. Stripe can keep the old secret active for a short transition and emits a signature for each active secret; Adyen warns that a new HMAC key can take time to propagate and advises temporarily accepting the previous key.13 A verifier should therefore support an explicit set such as active and retiring, expose which key verified each request, and alert if the old key is still in use near the deadline. After expiry, remove it from every verifier and secret store.
Compromise is not routine rotation. NIST key-management guidance calls for defined compromise-recovery roles and procedures, timely replacement, recording the incident and recovery actions, and an assessment of what the exposed key affected.5 In that path, a long overlap preserves the attacker’s access. Revoke the key, deploy the replacement, identify the period and endpoints exposed, and reconcile every consequential transition accepted during that window.
Ordering belongs to the state machine
Transport order is not business order. Stripe does not guarantee that events arrive in the order they were generated and advises retrieving missing objects through its API.3 Adyen tells consumers to check event timestamps and, where supplied, sequence numbers.4 Neither a signature nor a queue arrival time can turn an old transition into the current truth.
Model the consumer as a state machine. Store the provider’s object version or sequence when one exists. Reject transitions that move backwards. Where the provider offers no monotonic version, use the event as a prompt to fetch the object’s current state, then converge the local record to that state. A payment_failed notification arriving after payment_succeeded must not reopen the payment merely because it was dequeued later.
Ordering should be scoped to the smallest entity that requires it. Serialising every webhook through one worker protects order by destroying throughput and availability. Partition work by payment, order, or subscription identifier; allow unrelated entities to proceed concurrently; and send irreconcilable transitions to a quarantine queue with enough context for investigation.
Keep the public receiver small
A correctly signed payload is authenticated input, not harmless input. OWASP recommends syntactic and semantic validation for data from external partners and vendors because those feeds can still be malformed or compromised.6 The public endpoint should consequently do little: enforce request-size and content-type limits, preserve raw bytes, verify the signature and freshness, validate the envelope and allowed event type, durably insert it, and return a success response.
Move business processing to workers behind a queue. Stripe recommends asynchronous handling so delivery spikes do not overwhelm endpoint hosts, while Adyen advises acknowledging before lengthy processing and places unaccepted messages into a retry queue.34 The receiver can have permission to append to the inbox without permission to refund, fulfil, or alter a customer balance. Workers receive only the privileges required for their event family.
Network boundaries should reinforce that split. PCI Security Standards Council guidance describes segmentation as controls that separate systems with different security needs, using measures such as firewalls and configurations that prevent unwanted communication.7 Applied here, the internet-facing receiver should not gain a general route into the cardholder-data environment simply because it validates HMAC correctly.
Recovery is part of correctness
Provider retries buy time; they do not constitute a recovery plan. Public policies vary substantially: PayPal documents up to 25 delivery attempts over three days, Stripe retries live-mode events for up to three days, and Adyen can retain failed deliveries in a retry queue for up to 30 days.839 An outage can outlast a retry window. A handler bug can return success while applying the wrong transition. Manual replay can also race with automatic retries.
Recovery therefore needs three independent sources of truth: the durable inbox of what arrived, provider-side event history for what did not, and the provider’s current resource state for reconciliation. The runbook should define a last-known-good cursor, a backfill procedure, rate limits, quarantine rules, and a processing-code version. Stripe’s recovery guide explicitly warns that manually processed events may still be retried and recommends marking events as processing or processed before applying them.10
Test recovery with the awkward cases: the database commits but the HTTP response is lost; the old key verifies during rotation; two workers claim the same event; success arrives before failure; processing is broken for hours while the endpoint keeps returning 2xx; a compromised key signs plausible new event IDs. The system is ready only when replaying the test corpus converges to the same business state without repeating an external effect.
The signature remains essential. It keeps forged bytes outside. But a trustworthy webhook pipeline also needs time to reject captured bytes, memory to recognise repeated work, key discipline to survive rotation, state rules to absorb disorder, isolation to contain valid-but-dangerous input, and a ledger from which to recover. The first check is cryptographic. The rest is systems engineering.
Sources
“Verify HMAC signatures,” Adyen Docs (opens in a new tab).
“RFC 9421: HTTP Message Signatures,” Internet Engineering Task Force (opens in a new tab).
“Receive Stripe events in your webhook endpoint,” Stripe Documentation (opens in a new tab).
“Handle webhook events,” Adyen Docs (opens in a new tab).
“Input Validation Cheat Sheet,” OWASP Foundation (opens in a new tab).
“Integrate webhooks,” PayPal Developer (opens in a new tab).
“Troubleshoot webhooks,” Adyen Docs (opens in a new tab).
“Process undelivered webhook events,” Stripe Documentation (opens in a new tab).