When a ledger can rebuild its own past
Editing a damaged projection may fix one screen while dependent balances and reports retain the defect. A correction needs a reproducible shadow rebuild and a receipt.
8 min read
A ledger can rebuild its own past when its balances are outputs, not memories. Give a fresh projector the same immutable events, in the same order, under named logic and configuration, and it should derive the same accounts, balances and transaction states. That is replay as a capability: not a checkbox, but a result another operator can run, inspect and challenge.
The proof is the rebuild receipt. It identifies the event boundary, snapshot, schema and projector versions; reports what was processed, skipped or rejected; checks accounting and reconciliation invariants; and compares the rebuilt projection with the one serving production. Stripe’s public account of its Ledger makes the underlying idea concrete: balances, fund flows and data-quality controls are transformations of an immutable event log, and past state can be reconstructed by processing events to a chosen point.1 Trust comes from being able to repeat that claim under observation—especially when something has gone wrong.
The source is a sequence, not a balance
In a conventional state table, a balance of 42 says where the system ended. It may not say whether the route was one credit of 42, two credits and a reversal, or an incorrect write later patched in place. Event sourcing preserves the route: each accepted change becomes part of an ordered history, while query tables are derived views. A complete rebuild discards those views and runs the history against an empty state.2
For a payment ledger, an event needs enough stable business meaning to survive the software that first handled it. That normally includes an event ID, a business transaction reference, accounts, amount and currency, event type, effective time, recorded time, source, sequence and schema version. The write boundary must also define which order is authoritative. A timestamp alone is a weak tie-breaker when events can arrive late or share the same time; public ledger guidance therefore uses persisted account and transaction versions to identify exact historical states.5
The derived state can be disposable; the facts cannot. Materialized views exist because replaying a long history for every read is expensive. Snapshots shorten recovery by pairing a known state with a known event position, but they remain an optimization: the event stream is still the authority and a snapshot should itself be reproducible.3
Money adds useful invariants. A journal entry should balance, and credits and debits should balance per currency.45 Those checks do not prove that every real-world payment was recorded, but they make an important class of corruption impossible to hide inside a plausible total.
Determinism has an input budget
An immutable history is necessary for replay, not sufficient. Determinism requires the same initial state, the same ordered inputs and the same replay logic to produce the same result. Wall-clock reads, random identifiers, mutable global state, live database queries and external API calls can all send a replay down a different path; AWS’s replay guidance names each as a source of nondeterminism and records their results at a durable boundary instead.7
That rule is especially sharp in payments. If fee calculation used an exchange rate returned by another service, the historical rate or the service response must be captured as an input. Looking up today’s rate during replay reconstructs a different transaction. If processing sent a bank instruction, replay must not send it again. Fowler’s event-sourcing guidance treats outbound effects and historical external queries as explicit gateways because an outside system cannot infer that a call came from a rebuild.2
Retries need stable identity for the same reason. PayPal documents the ambiguous case where a capture request times out after the server has already captured the payment: retrying with the same request ID returns the prior operation’s status instead of capturing again.6 At ingestion, an idempotency key prevents one intent from becoming two facts. In projection, an event ID or sequence lets a consumer prove that a duplicate delivery did not move the balance twice.
Code evolves too. Old event shapes cannot be interpreted by accident. A replayable ledger either retains compatible handlers or applies named, tested upcasters that translate older schemas while leaving stored events unchanged.3 The projector build, event-schema set, rounding rules and configuration are therefore part of the replay input, even though they are not money events.
One history can answer two different questions
“Replay the ledger” is ambiguous until the operator names the question.
A forensic replay asks: What did the system believe at that time? It stops at a defined event position and uses the code, schemas and reference data valid for that historical execution. Its job is reproduction. This is the path for explaining a statement, tracing an incident or showing why a decision followed from the information then available.
A corrective replay asks: What should the derived state be after the defect is addressed? It may use fixed projection logic, an upcaster or new compensating events. Its job is repair. The original facts remain visible. Microsoft’s event-sourcing guidance notes that fixing application code does not fix bad historical events; correction may require compensating events or replay-time transformations.3 Square describes the accounting counterpart: incorrect money movement is repaired with a new correcting entry rather than an in-place update, preserving the mistake and the correction.4
Confusing these modes destroys evidence. Rebuilding old state with new rules and calling it a reproduction hides the change. Re-running old rules and calling the result repaired preserves the bug. A replay manifest should label the mode and record every departure from the original execution.
Make the rebuild leave a receipt
The useful unit of observability is a replay run, not a log line. Its manifest should record a unique run ID; start and end event positions; the snapshot ID and checksum, if used; projector and schema versions; configuration digest; start and finish times; counts of accepted, duplicate, skipped and rejected events; and an output digest. If the run stops, the receipt should identify the last committed position and the reason.
Then come the domain checks. Do entries balance per currency? Does each materialized account balance equal the sum of its effective entries? Are clearing accounts zero where the business flow says they should be? Do source-system or settlement-report references have matching ledger events? Stripe describes clearing, timeliness and completeness as separate data-quality questions; a missing or mis-keyed event can surface as a non-zero clearing balance rather than disappear inside an aggregate.1
The final check is a diff. Build into a shadow projection, compare it with the live view at the same event boundary, and classify every difference: missing input, duplicate input, order disagreement, schema incompatibility, nondeterministic dependency or projector defect. Zero difference is meaningful only when both sides consumed the same declared boundary.
Execution telemetry helps explain the run, but it is not the ledger history. W3C Trace Context standardizes identifiers that follow a request across distributed services, and OpenTelemetry can place trace and span IDs in logs so records from the same execution correlate.89 A replay run ID should join that trace context to the durable event IDs it processed. Trace IDs explain how this rebuild ran; event IDs establish which financial facts it rebuilt.
Repair is a promotion, not a mutation
Once a shadow rebuild passes its checks, repair becomes a controlled promotion. Freeze the input boundary. Produce the candidate projection. Review invariant failures and diffs. Reconcile material balances or external records. Then switch readers to the repaired view or advance the live projection from the verified checkpoint. The old projection and replay receipt remain available until the result is accepted.
This avoids the most tempting incident response: edit the number that looks wrong. An update may restore one screen while leaving dependent balances, reports and caches inconsistent. An append-only correction followed by deterministic derivation gives every downstream view the same repair path. Stripe’s published design similarly describes immutable-ledger correction as reverting and reprocessing prior operations, with proposed changes evaluated before commit.1
Financial-sector resilience guidance reinforces the distinction between restoration and proof. The EU’s Digital Operational Resilience Act requires checks and reconciliations after recovery, including when data is reconstructed from external stakeholders, to maintain consistency between systems.10 The Bank of England likewise identifies clean data and suitable reconciliation tools as important when recovering from a loss of data integrity, warning that uncertainty about accuracy can lengthen recovery and disrupt payment flows.11
A ledger has not earned “replayable” because its architecture diagram contains an event log. It earns the word when a routine exercise can rebuild a declared past, produce a verifiable receipt, expose every divergence and support a correction without erasing what happened. At that point, replay stops being a feature label. It becomes evidence that the ledger can be trusted when trust is hardest: during explanation, recovery and repair.
Sources
- “Ledger: Stripe’s system for tracking and validating money movement.” (opens in a new tab) Stripe Developer Blog.
- “Event Sourcing.” (opens in a new tab) Martin Fowler.
- “Event Sourcing pattern.” (opens in a new tab) Microsoft Azure Architecture Center.
- “Books, an immutable double-entry accounting database service.” (opens in a new tab) Square Developer.
- “How to Scale a Ledger, Part V: Immutability and Double-Entry.” (opens in a new tab) Modern Treasury.
- “Idempotency.” (opens in a new tab) PayPal Developer.
- “Determinism during replay.” (opens in a new tab) AWS Durable Execution SDK Developer Guide.
- “Trace Context.” (opens in a new tab) World Wide Web Consortium.
- “OpenTelemetry Logging.” (opens in a new tab) OpenTelemetry.
- “Regulation (EU) 2022/2554 on digital operational resilience for the financial sector.” (opens in a new tab) European Parliament and Council of the European Union.
- “Operational resilience in a macroprudential framework.” (opens in a new tab) Bank of England.