The fastest payment API can still lose the benchmark

An API can win the median-latency race and still be a worse payment API. If its slowest one per cent of requests cross the checkout deadline, if a timeout leaves the caller unable to tell whether money moved, or if an upgrade changes the meaning of a webhook, a quick typical response has hidden the costs that matter.

A useful benchmark therefore does not ask only, “How fast did the endpoint reply?” It asks whether the system returned the right state, whether an error told the caller what to do, whether a retried mutation remained one financial operation, whether old and new versions behaved compatibly, and how long an ambiguous payment took to become certain. Median latency stays on the report. It simply loses its veto over everything else.

Benchmark the payment decision, not the HTTP exchange

The unit under test should be a payment operation as experienced by an integrating system. For a synchronous authorization, that begins when the client is ready to send a complete request and ends when the client has a response it can safely map to a business action. For a challenge flow, it includes the hand-off and return of authentication. For an asynchronous method, it has at least two clocks: time to a useful intermediate state and time to a trustworthy terminal state.

That distinction is not semantic decoration. EMV 3-D Secure defines separate frictionless and challenge flows, with the latter requiring additional cardholder interaction.5 Adyen documents Received and Pending as intermediate payment results and warns that a payment’s status can change after the synchronous result code is returned.7 A benchmark that stops its timer at the first 2xx can reward an API for answering before the payment question has been resolved.

The comparison also needs a manifest. Pin the client region, transport settings, connection reuse, SDK and API versions, request timeout, retry policy, concurrency, arrival-rate model, currency, amount class, payment method, authentication path and operation mix. Run every candidate from the same hosts through the same network path. Rotate run order so a daily traffic pattern does not always favour the same candidate. Retain the request schedule, sanitized responses, event transcript and raw latency histogram for every run.

Separate semantic testing from load testing. Provider sandboxes are valuable for scripted approvals, declines, authentication and refunds, but they are simulations. Stripe states that its test environments create simulated objects without card-network or payment-provider processing, and explicitly says not to load-test the integration in that environment because rate limits may apply.4 Use sandboxes for conformance and fault-path coverage. Measure capacity only in an environment and at a rate the provider has authorized. A published result should identify the environment instead of quietly treating sandbox speed as production speed.

Finally, predeclare what “correct” means. A response is not correct merely because it matches a schema. It is correct when the observed provider state, the local state transition and the eventual financial record agree with the scenario’s expected outcome. Without that oracle, a benchmark can precisely time the wrong result.

The workload must be declared with the same care as the oracle. Construct it from named cells rather than one bag of requests: operation, payment method, authentication path, outcome class, client region, connection state and offered load. Run enough observations inside each cell to report it independently. Only then combine cells using weights taken from the intended production mix. Otherwise a candidate can appear faster because it received more status queries while another received more challenged authorizations.

Keep two views of the result. The standardized view gives every candidate the identical fixed mix and answers the procurement question under one controlled workload. The workload-adjusted view reweights the same per-cell observations for a particular integration and answers the engineering question for that business. Never rerun a candidate with a more favourable mix and call the totals comparable.

This design also blocks a compositional reversal in the headline number: an aggregate can move even when none of its underlying journey classes improved, simply because the class proportions changed. The defense is visible strata, fixed weights and paired run windows. Publish all three. A reader should be able to recompute the aggregate after excluding an irrelevant method or changing the regional mix without touching the raw experiment.

Median latency is a diagnostic, not a verdict

The median describes the typical observation. It says nothing about how far the tail extends. Dean and Barroso’s “The Tail at Scale” explains why temporary high-latency episodes that appear unimportant in smaller systems can dominate a larger service as complexity and utilization rise.1 Google’s SRE guidance makes the measurement consequence explicit: high percentiles expose plausible worst-case behaviour, while the 50th percentile emphasizes only the typical case.2

Payment traffic makes the distinction operational. A request that completes after the caller’s deadline is not merely “slow.” It may trigger a retry while the first attempt is still running. If the original mutation later succeeds, latency has crossed into correctness and recovery. The benchmark should therefore report p50, p95, p99 and p99.9, but also the fraction that missed each declared client deadline. Timed-out observations remain in the denominator; deleting them because no response arrived manufactures a faster distribution.

Use an open-loop load generator for the main capacity curve: requests arrive according to the declared schedule rather than waiting for the previous request to finish. A closed-loop generator can commit coordinated omission. During a stall it stops generating the requests that would have arrived, so it records too few bad experiences. Friedrich, Wingerath and Ritter demonstrate that open- and closed-system load models can produce substantially different percentile spectra and conclude that the load-generation model cannot be corrected reliably after the run.3

One latency distribution is still too coarse. Keep authorization, capture, reversal, refund and status-query histograms separate. Split new connections from reused connections. Split frictionless, challenged and asynchronous journeys. Report the offered request rate next to the achieved rate, because a low latency achieved by silently falling below the target load is not a successful run. Plot percentiles as load increases; the knee where queues form is more informative than one lightly loaded number.

Measure from the caller, not only at the provider edge. Record DNS, connection and TLS setup where the client can observe them, then time to first byte and time to the complete body. Keep provider request identifiers so an anomalous observation can be investigated. The externally visible duration is the score; the phases are diagnostic evidence. This prevents a benchmark from subtracting the very network and serialization costs the integration must pay.

Tail results require uncertainty, not just extra decimal places. Repeat the complete run in independent windows, publish the number and duration of repetitions, and show the distribution across runs. A single p99 from a small sample is fragile: by definition, only a small part of that sample determines it. The benchmark should refuse to rank two candidates on a tail metric when the repeated results do not establish a stable ordering.

Correctness has to be a gate

Payments are stateful protocols built from linked operations. Visa Acceptance describes capture as a follow-on transaction to authorization, an authorization reversal as the operation that releases the hold, and a void as cancellation of a capture or credit that has not yet been submitted to the processor.6 These transitions cannot be reduced to “request succeeded.” A fast capture against the wrong authorization is not a near miss.

Build the correctness suite as journeys with explicit preconditions, actions and invariants. It should cover a straight authorization, authorization followed by full capture, partial capture plus reversal of the unused amount, duplicate capture, full and partial refund, authentication followed by issuer decline, a challenge abandoned by the cardholder, and a payment that remains pending before completing. Add malformed amounts, unsupported currencies, expired credentials and stale resource versions. The exact set should reflect the production payment-method mix and be published with the result.

Each journey asserts both state and value. Captured amount cannot exceed authorized amount. Total successful refunds cannot exceed captured amount. A reversal cannot increase the amount available to capture. One business attempt cannot create two successful financial effects. A terminal success cannot later be overwritten by a delayed intermediate message. If manual capture is enabled, authorization alone cannot satisfy an oracle that requires captured funds. These are benchmark rules, not values inferred from whichever API is under test.

Score four kinds of semantic failure separately. A false positive says the payment is usable when it is not. A false negative reports failure after the financial effect occurred. A transition error reaches an impossible local state. A value error loses or changes amount, currency or linkage. False positives and duplicate financial effects should be disqualifying gates because a weighted average would let excellent latency compensate for an unsafe answer.

The test harness must observe more than the synchronous body. Query the resource after each journey, consume its events, and reconcile the final transaction record where the environment exposes one. Adyen’s result-code documentation itself advises against using the synchronous result code to update an order-management system because status can change, and points to the authorization webhook for the updated payment state.7 That is why correctness needs an observation window long enough for the scenario to settle.

This gate also makes the latency comparison fairer. A candidate does not earn speed points for returning an optimistic state that another candidate withholds until it has stronger evidence. First establish that both responses authorize the same business decision. Then compare how quickly they deliver it.

Error quality is executable information

HTTP status is necessary and insufficient. Visa Acceptance documents that 201 confirms creation of a transaction resource, yet the transaction itself can be under review, declined or failed.8 A generic transport classifier that maps every 2xx to paid will be fast, deterministic and wrong.

The benchmark should ask whether an error can drive a safe next action without parsing prose. RFC 9457 defines a machine-readable problem-details format because HTTP status codes alone do not carry enough information for API consumers; its model includes a problem type and can include an identifier for the specific occurrence.9 PayPal’s public error format illustrates domain detail with an error name, debug identifier, affected field, location, issue and documentation link.10 These are observable properties that can be scored.

Inject failures at different boundaries: invalid request fields, failed authentication, rate limiting, provider-side unavailability, issuer decline, client timeout before any response, and connection loss after the request body was sent. For each response, evaluate whether it contains a stable machine code, identifies the invalid field when applicable, carries a correlation reference, distinguishes a terminal payment decision from a transient technical fault, and states or documents whether retry is safe. Human-readable text helps operators, but client code should not have to classify adjectives.

Retry guidance must be tested, not assumed. Visa Acceptance publishes decline categories that distinguish an issuer that will never approve from categories that may be retried; one category instructs the caller to revalidate payment data first.11 An API that collapses those outcomes into declined transfers policy into guesswork. Conversely, a 500 labelled “temporary” is incomplete if the caller cannot tell whether the mutation took effect.

The result is an actionability score. Give full credit when the harness can deterministically choose one of: correct the request, re-authenticate, ask the customer for different details, retry the same logical operation, wait and query, or stop. Give partial credit when documentation supplies the rule but the response omits it. Give none when the only safe automation is to open a support case. Error latency may be reported, but a quick ambiguous error does not improve the score.

Version stability belongs in the timed course

API quality includes the cost of remaining integrated. Stripe documents that its API version controls accepted parameters, response properties and webhook behaviour; major releases can contain backward-incompatible changes, while its monthly releases within a named release are backward-compatible.12 Adyen likewise distinguishes breaking from non-breaking changes and notes that the classification can depend on the API family; for its newer APIs, adding a response field is non-breaking, while the same change is classified differently for listed classic APIs.13

A benchmark can turn change risk into an experiment. Capture sanitized fixtures for every synchronous response and event in the correctness suite. Run the consumer against the pinned current version, the target version and deliberately extended payloads. Add unknown enum values, new event types, reordered object fields, longer opaque identifiers and optional fields. Then replay old events through the new consumer and new events through the compatibility layer intended for the upgrade.

Measure the failures, not the changelog length. Count schema-deserialization failures, changed normalized outcomes, lost fields required for reconciliation, code paths that depend on human-readable error strings, and fixtures that produce a different business decision. Record engineering work only under a defined protocol: same client abstraction, same test suite, same upgrade boundary. Otherwise “hours to upgrade” measures team familiarity more than API stability.

Asynchronous payloads need their own version trial. Stripe notes that changing the version for a specific API request does not change the version used to create an event, and that old event objects retain the structure determined when they were created.14 The benchmark should therefore replay a mixed-age event archive after an upgrade. Passing only freshly generated synchronous fixtures tests the easiest half of the contract.

Version stability should reward explicit pinning, a machine-readable or diffable change record, coexistence long enough for a staged migration, and a tested rollback path. It should penalize silent decision drift even when JSON deserialization succeeds. Compatibility is not “the parser did not throw.” It is “the same evidence still produces the intended payment action.”

Recovery starts where the stopwatch would normally stop

The defining payment failure is an unknown outcome. PayPal’s idempotency documentation gives the concrete case: a capture request times out after the server captures the payment; retrying with the same request identifier returns the current result without capturing again.15 The first duration ended in a timeout. The payment operation did not end.

Idempotency semantics also differ in details a checkbox cannot express. Stripe stores the first status code and body for a key, including 500 responses, compares replayed parameters, and permits keys to be removed after they are at least 24 hours old.16 Adyen says its keys are scoped to a company account for at least seven days and are not checked for duplicates across regional endpoints; it also exposes a transient-error signal for certain safe retries.17 A benchmark must test the documented namespace, retention, parameter-matching and concurrency behaviour of the exact operation. “Has idempotency” is not a result.

Run a recovery suite through a programmable proxy. Let the provider receive the full mutation, then sever the response. Retry concurrently with the same operation identity. Replay with changed parameters. Delay the retry beyond the documented deduplication window. Switch regional endpoints where the architecture permits it. After every case, query state and count financial effects. The primary measures are duplicate-effect rate, false-terminal-state rate, time to certainty and number of manual decisions required.

Then break the event path. Hold notifications while synchronous operations continue, release them out of order, deliver the same event twice, and restore the endpoint after a backlog forms. Stripe explicitly says event delivery order is not guaranteed and an endpoint may receive the same event more than once.14 Measure backlog drain time, duplicate processing, illegal state transitions and the time at which every affected payment converges on the queried provider state.

Recovery needs a final evidence layer. A webhook proves that an event was emitted and received; it does not by itself prove that every internal record matches settlement. Public settlement documentation shows why a separate comparison exists: Adyen’s Settlement details report contains settled payments and transaction-level entries intended for reconciliation.18 Where equivalent reporting is available, the benchmark should join each test operation to its final financial record and report missing, duplicated or mismatched joins.

Do not hide this work under “operational maturity.” It is measurable API behaviour. A provider that takes 150 milliseconds longer in the normal case but turns an ambiguous timeout into one safe replay may impose less end-to-end delay, less code and less risk than the median winner.

A scorecard that allows speed to lose

Weighted scores can conceal catastrophes, so apply eligibility gates first. Disqualify a candidate for any reproducible duplicate financial effect, false terminal success, amount or currency corruption, or absence of a safe path from an injected ambiguous outcome. A failed gate remains visible beside the raw evidence; it is not converted into a small deduction.

For candidates that pass, one defensible 100-point rubric is:

| Dimension | Points | What earns them | | --- | ---: | --- | | Semantic correctness | 30 | Correct states, amounts and linkages across the published journey suite | | Recovery | 25 | Safe retries, rapid convergence, replayable events and reconciled final records | | Tail latency | 20 | Deadline success, p95, p99, p99.9 and stability as offered load rises | | Error actionability | 15 | Machine codes, outcome certainty, field detail, correlation and safe next action | | Version stability | 10 | Pinned behaviour, fixture compatibility, explicit deltas and reversible migration |

Median latency is still disclosed and can take a small share of the latency dimension. It cannot offset a poor deadline-success rate, and the entire latency dimension cannot offset a correctness-gate failure. This is the promised reversal in mathematical form.

Inside the 20 latency points, make the demotion explicit. A sample allocation is two points for p50, four for p95, four for p99, two for p99.9, six for completion within the business deadline and two for maintaining the target arrival rate without tail collapse. The exact thresholds come from the declared use case. The structure matters more than these illustrative numbers: typical speed receives credit, but deadline reliability and the tail receive most of it.

Normalize each continuous measure against a target declared before testing, not against the best candidate observed afterward. For example, score deadline completion against the checkout’s actual timeout budget and recovery against the maximum tolerable time in unknown. Candidate-relative normalization makes a system’s score change when an unrelated competitor enters the field. Target-relative scoring keeps the benchmark tied to the integration’s requirements.

Publish dimension scores before the total. Different businesses can legitimately change the weights: digital fulfilment may make false success especially costly, while an in-person flow may have a tighter interaction deadline. They should not change the gates. No payment context makes a duplicate capture acceptable because its p50 was excellent.

Reproducibility statement

A benchmark report should ship with the scenario definitions, expected state transitions, load schedule, random seeds, harness revision, sanitized fixtures, API and SDK versions, client regions, timestamps, sample counts, retry rules, timeout budgets, raw histograms and scoring code. It should identify excluded payment methods and every test that the provider environment could not express. Another team may not be able to recreate identical issuer decisions, but it should be able to inspect every transformation from observation to score.

The conclusion must remain as narrow as the workload. Sandbox results establish behaviour against simulated cases, not live acquiring performance; Stripe’s documentation explicitly separates simulated test objects from live requests processed by card networks and payment providers.4 A card-only benchmark does not rank bank-transfer APIs. One client region does not establish global latency. One pinned version does not predict every future migration. These are boundaries, not footnotes.

Within those boundaries, the winning question is clear. Which API most often delivers a correct, actionable payment result before the business deadline—and, when it cannot, preserves one financial intent and returns the system to certainty? The answer may also have the lowest median. But if it does not, the faster API has not been robbed by the benchmark. It has finally been measured as a payment API.

Sources

1. “The Tail at Scale (opens in a new tab).” Google Research / Communications of the ACM.
2. “Service Level Objectives (opens in a new tab).” Google, Site Reliability Engineering.
3. “Coordinated Omission in NoSQL Database Benchmarking (opens in a new tab).” Steffen Friedrich, Wolfram Wingerath and Norbert Ritter, Gesellschaft für Informatik.
4. “Testing (opens in a new tab).” Stripe Documentation.
5. “Introduction: EMV 3-D Secure White Paper (opens in a new tab).” EMVCo.
6. “Payments (opens in a new tab).” Visa Acceptance Solutions.
7. “Result codes (opens in a new tab).” Adyen Docs.
8. “Transaction response codes (opens in a new tab).” Visa Acceptance Developer Portal.
9. “RFC 9457: Problem Details for HTTP APIs (opens in a new tab).” Internet Engineering Task Force.
10. “API responses (opens in a new tab).” PayPal Developer.
11. “Declined Authorizations (opens in a new tab).” Visa Acceptance Solutions.
12. “API upgrades (opens in a new tab).” Stripe Documentation.
13. “Versioning (opens in a new tab).” Adyen Docs.
14. “Receive Stripe events in your webhook endpoint (opens in a new tab).” Stripe Documentation.
15. “Idempotency (opens in a new tab).” PayPal Developer.
16. “Idempotent requests (opens in a new tab).” Stripe Documentation.
17. “API idempotency (opens in a new tab).” Adyen Docs.
18. “Settlement details report (opens in a new tab).” Adyen Docs.