Your authorization lift may be a mix shift

An aggregate authorization rate can rise even when routing has improved nothing. If more payments now come from issuers, countries or transaction types that were already easier to approve, the portfolio average goes up. The route gets the applause; the mix did the work.

The reverse is possible too. A route can improve approval within every meaningful segment while the headline rate falls because the portfolio moved toward harder traffic. Aggregate data producing a different conclusion from disaggregated data is a documented statistical failure mode, not a payments curiosity.1 A before-and-after authorization chart therefore cannot, by itself, identify routing impact. It measures the combined movement of route performance and the traffic weights applied to it.

The rate is a weighted average

Let each segment \(s\) be a defensible slice of traffic: perhaps issuer country × card brand × transaction type × authentication state. Its authorization rate is \(ps\), and its share of attempts is \(ws\). The aggregate rate is:

A = \sum_s w_s p_s

Both terms can move. Routing may change \(ps\). Marketing, billing cycles, geographic expansion, outages, fraud controls or customer behavior may change \(ws\). A dashboard showing only \(A\) collapses these two processes into one number.

Consider a deliberately simple portfolio. Domestic-issued cards authorize at 90%; cross-border cards authorize at 60%. In week one, the mix is 80% domestic and 20% cross-border, producing an 84% aggregate rate. In week two, the segment rates remain exactly 90% and 60%, but the mix becomes 90% domestic and 10% cross-border. The aggregate rises to 87%. The reported lift is three percentage points. The route effect is zero.

This does not require a full reversal of the sort commonly called Simpson’s paradox. Any change in weights can exaggerate, conceal or manufacture a change in the average. The classic Berkeley admissions analysis found that aggregate data showed a clear but misleading pattern that changed when the relevant decision-making units were examined separately.1 In payments, those decision-making units are not university departments. They are issuer and transaction contexts with different baseline approval probabilities.

The first research question after a reported lift should therefore be: lift on which fixed population? If the answer is “all attempts before versus all attempts after,” the measurement has not yet separated performance from composition.

Payment mix moves on several axes at once

Issuer is not a decorative reporting field. The issuing bank’s systems decide whether to authorize a card payment and may evaluate account balance, spending behavior, card data and fraud signals.2 That makes issuer composition an immediate candidate confounder. A promotion that attracts more customers from one bank, a payroll date that changes available balances, or an expansion that adds a new issuing market can move authorization without a routing change.

Geography is entangled with more than issuer identity. Public routing documentation describes decisions based on issuing country, currency, amount, device, card type, geographic proximity and card-brand performance.3 Those are useful routing inputs precisely because payment contexts differ. They are also measurement variables. If a treatment route receives a different distribution of them, comparing its raw rate with another route answers “which traffic did each route receive?” as much as “which route performed better?”

Transaction semantics matter as well. Entry mode, point-of-sale versus ecommerce interaction, merchant category and counterparty country are among the contextual fields that public payment documentation recommends for investigating authorization performance.4 A first customer-initiated payment is not interchangeable with a subsequent merchant-initiated charge. A manually entered transaction is not interchangeable with a credential presented through a digital wallet. Pooling them assumes away differences the authorization system is built to inspect.47

Authentication can alter the population that reaches authorization. EMVCo defines frictionless and challenge flows for EMV 3-D Secure: an issuer can accept the authentication without challenging the cardholder after real-time risk assessment, or require additional interaction when the transaction is considered higher risk or needs cardholder confirmation.5 Under PSD2, strong customer authentication requirements and their exemptions also depend on the payment context; the payer’s payment service provider makes the ultimate decision on accepting or applying an exemption.6 A change in geography or 3DS policy can therefore change both who completes authentication and what evidence accompanies the authorization.

Credential mix is another layer. EMV payment tokens replace the primary account number with an alternative value and can be constrained to a merchant, device or payment scenario.7 Tokenized and non-tokenized traffic are not merely two labels attached after routing. They can carry different security evidence and route eligibility. If token adoption rises during a routing rollout, the aggregate result combines two interventions unless the analysis separates them.

None of this proves that issuer, geography, authentication or credential type caused a particular observed change. It establishes why a portfolio average is underidentified: several plausible weights and mechanisms moved at the same time.

Routing changes the sample it is judged on

The measurement problem becomes sharper when the router selects traffic rather than receiving it randomly. A routing engine may use issuing country, currency, amount, card type, device, historical outcomes, latency or cost to choose a processor.3 A route’s observed population is then a product of the policy being evaluated.

Suppose route B is enabled only for domestic debit cards below a certain amount, while route A continues to receive the rest. B may immediately show the higher raw authorization rate. That is not evidence that moving the remaining traffic to B would reproduce the result. B has not been tested on the remaining traffic.

Fallbacks produce the opposite distortion. If route A receives first attempts and route B receives only unresolved or retryable failures, B begins with a negatively selected sample. Comparing A’s first-attempt rate with B’s fallback rate is structurally unfair. Giving B credit for an order recovered after A’s decline creates another ambiguity: it measures sequence performance, not the isolated effect of B as the initial route.

Even “eligible traffic only” can become circular if eligibility is computed by the new policy. The treatment may remove hard transactions from its own denominator, send them elsewhere, or suppress attempts entirely. The population must be defined at a point before the route decision, using rules that are shared by control and treatment. Otherwise the intervention chooses both the treatment and the test set.

This is why route dashboards are useful for operations but weak as causal evidence. They describe what happened after selection. They do not supply the counterfactual: what would have happened to the same payment intent on the other eligible route.

Audit balance before looking at approvals

A mix-shift investigation should begin with counts and weights, not decline reasons. Build a balance table for the comparison periods or experimental arms. For every predeclared segment, show payment-intent count, attempt count and portfolio share. Then show the change in share. A country whose authorization rate is unchanged but whose weight doubled belongs near the top of the explanation, not in an appendix.

The table needs intersections, not only separate marginals. “Issuer-country mix was stable” and “merchant-initiated mix was stable” do not imply that issuer country × initiation type was stable. One arm may have gained first payments from a country while losing recurring payments from the same country, leaving both top-line shares unchanged. Since authorization behavior can depend on the combined transaction context, marginal balance can conceal joint imbalance.

There is a practical limit. Cross every issuer, country, brand, currency, amount, credential and authentication value, and most cells will be too sparse to interpret. The segmentation hierarchy should therefore be frozen before outcomes are read. Start with variables that exist before routing, have a credible relationship with authorization, and can be measured consistently across providers. Preserve a small set of business-critical intersections. Pool the rest using a rule based on support, not on whether their observed lift is flattering. Post-stratification research is explicit that many small strata can reduce rather than improve precision.13

Overlap is as important as balance. If route B never receives a particular issuer-country cell, historical data cannot reveal B’s effect in that cell. Reweighting B’s observed traffic to include it would require an outcome that was never observed. Report that cell as unsupported. The honest target becomes the shared-support population, while the excluded traffic remains a separate coverage question.

Time deserves the same treatment as geography. A sequential rollout compares different calendar periods, so any concurrent shift enters the estimate: issuer availability, a campaign, a billing run, an authentication-policy change or a provider incident. Mastercard’s rules explicitly classify issuer unavailability, malfunction and timeout as authorization failures for performance measurement.10 A period with more such failures is not comparable merely because it contains the same card brands.

Contemporaneous randomization is the strongest protection because control and treatment experience the same hours.12 If deployment must ramp by region or merchant group, retain a concurrent holdout in every feasible phase. At minimum, compare like time blocks—hour of week and rollout phase—and publish traffic weights for each. A single “four weeks before versus four weeks after” line invites calendar effects to masquerade as route effects.

Finally, do not balance on variables created by the treatment. Executed route, fallback position, challenge result, retry count and final decline class occur after assignment or can be changed by policy. Conditioning on them breaks the original comparison; CONSORT’s experiment guidance similarly warns that post-randomization exclusions compromise randomization and turn a per-protocol view into an observational comparison.16 Use these fields to explain mechanisms after the primary estimate, not to decide which records count toward it.

Fix the denominator before estimating lift

“Authorization rate” can hide a second mix problem: the mix of first attempts, retries and duplicate observations. One public Visa definition calculates approved authorizations over total authorization attempts using the first attempt of a unique transaction.8 That qualifier matters. An attempt-level warehouse that includes every retry is estimating a different quantity.

Take 100 payment intents. Eighty are approved on the first attempt. The remaining 20 are retried, and eight of those are approved. First-attempt authorization is 80%. Eventual payment-intent success is 88%. Attempt-level authorization is 88 approvals divided by 120 attempts, or 73.3%. Nothing about the payment behavior changed across these three statements; only the unit and denominator did.

An orchestrator that suppresses low-value retries could raise the attempt-level rate while recovering fewer customer orders. One that adds disciplined retries could lower the attempt-level rate while increasing eventual success. Neither result is contradictory. The metrics answer different questions.

Retries also cannot be treated as generic extra samples. Network decline and advice codes carry different meanings, and their interpretation can vary by card network.2 Visa’s public response tables distinguish conditions such as insufficient funds, additional authentication required, duplicates, issuer or switch unavailability and system malfunction.9 Mastercard’s public rules separately define authorization failure rates caused by issuer unavailability, malfunction or timeout.10 A retry population is therefore a mixture of financial declines, authentication requirements, technical failures and indeterminate outcomes. Route performance will look different when that mixture changes.

The measurement contract needs at least three explicit metrics. First-attempt authorization asks whether the initial authorization was approved. Attempt authorization describes processing efficiency across all submitted attempts. Payment-intent success asks whether one commercial intent eventually reached the defined successful state within a fixed window. Each record needs a durable intent identifier, attempt sequence, assigned route, executed route, response class and final outcome so the three denominators can be reconstructed.

Authorization should also remain distinct from checkout conversion. Public payment documentation separates the share of shoppers who complete the offered payment flow from the authorization result after the acquirer processes the transaction.11 A 3DS challenge that a shopper abandons may never enter an authorization denominator. Reporting authorization lift alone can therefore miss deterioration earlier in the journey.

Split traffic before the router chooses

The cleanest estimate comes from a randomized experiment on traffic that is eligible for both policies. Identify the candidate population before routing. Randomly assign each payment intent to control policy or treatment policy, then let the assigned policy choose among its permitted routes.

The randomization unit should match the commercial question. If the question is whether a policy helps a customer payment succeed, assignment belongs on the payment intent, not on individual attempts. All attempts for that intent should remain in the same experimental arm. Controlled-experiment guidance emphasizes defining an experimentation unit and giving it a consistent treatment throughout the experiment.12 Attempt-level randomization would allow a retry to cross arms, contaminating both the causal comparison and the attribution of recovery.

Randomization should occur only where both policies are genuinely feasible. Transactions that only one route can process belong in an operational coverage report, not in the head-to-head estimate. Within the shared eligible set, assignment can be blocked on a small number of pre-treatment variables with strong baseline relationships to approval: issuer country or a stable issuer grouping, card brand, customer- versus merchant-initiated status, authentication state, token state and perhaps coarse amount or currency bands.

Blocking protects the experiment from a chance pile-up of hard traffic in one arm and makes the comparison legible inside important cells. It should not create thousands of sparse combinations. Research on post-stratification shows that adjustment can reduce variance when a fixed baseline covariate is prognostic, but can hurt precision in small samples when there are too many strata or the chosen variable is poor.13 More recent work also cautions that regression adjustment designed for complete randomization is not automatically efficient under stratified designs.14 “Control for everything” is not a statistical plan.

The primary readout should be intention-to-treat: compare outcomes by assigned policy, even when the intended route was unavailable and the system executed a permitted fallback. This preserves the benefit of random assignment and evaluates the policy as deployed.16 A secondary per-protocol view can explain execution failures, but it should not replace the assigned-arm result; excluding crossovers after the fact selects observations using a post-assignment event.

For each arm, report payment-intent success and first-attempt authorization on the same eligible population. Add attempt count per intent, technical failure rate, latency, authentication completion, processing cost and later risk outcomes as guardrails. A routing policy that gains approvals by generating materially more attempts, more challenge abandonment or unacceptable downstream risk has not produced a free lift.

Run an A/A test before trusting a small effect. Send both arms through equivalent policy and verify allocation, event joining, denominators and confidence-interval behavior. A/A tests are specifically used to assess variability and exercise the experimentation system when no treatment difference should exist.12 In payments, they are also a practical way to reveal duplicate attempts, missing terminal outcomes and assignment leakage before those defects acquire a business narrative.

Decompose every before-and-after result

Randomization is not always immediately available. A historical rollout can still be made more honest by decomposing the aggregate change instead of calling all of it lift.

For each predeclared segment, calculate its weight and approval rate before the change, \(w{0s}\) and \(p{0s}\), and after it, \(w{1s}\) and \(p{1s}\). The aggregate change has an exact symmetric decomposition:

A_1-A_0 =
\sum_s \frac{w_{1s}+w_{0s}}{2}(p_{1s}-p_{0s})
+
\sum_s \frac{p_{1s}+p_{0s}}{2}(w_{1s}-w_{0s})

The first term is the within-segment rate contribution: did authorization change while applying the average portfolio weights? The second is the mix contribution: did the portfolio move toward segments with higher or lower average rates? Together they reproduce the raw movement exactly.

Also publish a standardized rate by applying both periods’ segment rates to one frozen reference mix. The reference can be the pre-period portfolio, the pooled experiment population or another explicitly governed distribution. Its identity matters: standardizing to last quarter answers a different business question from standardizing to today’s eligible traffic. Do not quietly change the reference when the result becomes inconvenient.

These methods improve description, not causality. If an unobserved factor changed within a segment at the same time as routing, the “rate contribution” still contains both. Segmentation by issuer country does not control for an issuer-level policy change; an issuer grouping can hide different banks; and a wide amount band can conceal an internal shift. Historical standardization is therefore a diagnostic and a bridge to experimentation, not a substitute for a randomized counterfactual.

Use adjustment for precision, not storytelling

Once assignment is randomized, pre-treatment covariates can improve precision without redefining the target population. CUPED, a control-variate method developed for online experiments, uses pre-experiment data correlated with the outcome to reduce metric variance; its authors report roughly 50% variance reduction in their Bing application.15 The transferable lesson is not that every payments experiment will halve its variance. It is that information measured before treatment can make a noisy comparison more sensitive without borrowing from post-treatment outcomes.

Suitable payment covariates might include a merchant’s or issuer segment’s pre-period authorization rate, provided it is calculated before the experiment and at a grouping with enough support. The analysis must preserve treatment interactions or the design-specific estimator required by the randomization scheme. Post-stratification research finds that a fully saturated approach can be comparable with blocking, while warning that sparse or poorly chosen strata carry a precision cost.1314

Predeclare the primary metric, analysis population, segment hierarchy, retry window, reference mix and estimator. Controlled-experiment guidance warns that choosing comparisons after seeing results increases the chance of apparently significant findings by chance.12 The payments version of this mistake is familiar: slice by issuer, country, brand, amount, device, credential, weekday and route sequence until one green cell appears.

Confidence intervals belong beside every lift estimate. So do segment counts and weight changes. A standardized improvement driven by one tiny issuer cell is not equivalent to a broad effect across the portfolio. Segment estimates should be treated as heterogeneous evidence—with multiplicity and power limits—not as independent promises.12

What a convincing readout looks like

Imagine a rollout whose aggregate first-attempt authorization rises by 1.7 percentage points. The decomposition assigns 1.4 points to a larger share of domestic debit traffic and 0.3 points to within-segment rate changes. A fixed-mix estimate is +0.3 points with a confidence interval spanning zero. A randomized eligible-traffic holdout shows +0.2 points, also imprecise. The accurate conclusion is not “routing delivered 1.7 points.” It is “the portfolio improved by 1.7 points, mostly because its mix became easier; route lift was not established.”

Now invert the evidence. Aggregate authorization rises 1.2 points; the fixed-mix estimate is +1.1; the randomized intention-to-treat estimate is +1.0 with a confidence interval that excludes zero; gains appear in several adequately sized issuer-geography groups; retry volume, latency, authentication completion, cost and risk guardrails remain within their predeclared bounds. That is a credible routing result. The three views do not need to match exactly, but they tell the same causal story.

The readout should still preserve the raw aggregate. Mix is commercially real: a portfolio with more easy traffic does collect more approvals. The mistake is not reporting that outcome. The mistake is assigning its cause to routing.

The standard of proof

An authorization chart becomes research only when it states what remained fixed. The minimum defensible package is a stable payment-intent denominator, a pre-route eligible population, raw and fixed-mix rates, issuer and geography weight changes, retry-aware outcomes, and—where feasible—a randomized policy comparison.

This standard makes some apparent wins smaller. It will also recover real improvements that a worsening mix had buried. Both are useful. Orchestration should be credited for changing the probability of approval for comparable traffic, not for being present when the traffic changed.

Your authorization lift may indeed be routing lift. Until the weights are held still, it may just be a mix shift.

Sources

1. “Sex Bias in Graduate Admissions: Data from Berkeley” (opens in a new tab), P. J. Bickel, E. A. Hammel and J. W. O’Connell, Science, American Association for the Advancement of Science.
2. “Card declines” (opens in a new tab), Stripe Documentation.
3. “Intelligent payment routing: A guide for businesses” (opens in a new tab), Stripe.
4. “Authorization rates and troubleshooting” (opens in a new tab), Adyen Docs.
5. “Optimising Online Payment Authentication with EMV® 3-D Secure” (opens in a new tab), EMVCo.
6. “2018_4089: Obligatory nature of the SCA and exemption based on transaction risk analysis” (opens in a new tab), European Banking Authority.
7. “EMV® Payment Tokenisation” (opens in a new tab), EMVCo.
8. “Token Management Service: Drive innovation and growth with a flexible, robust tokenization solution” (opens in a new tab), Visa Acceptance Solutions.
9. “Request and Response Codes” (opens in a new tab), Visa Developer Center.
10. “Transaction Processing Rules” (opens in a new tab), Mastercard.
11. “What is the difference between authorization and conversion rates?” (opens in a new tab), Adyen Help.
12. “Practical Guide to Controlled Experiments on the Web” (opens in a new tab), Ron Kohavi, Randal M. Henne and Dan Sommerfield, KDD 2007, Association for Computing Machinery.
13. “Adjusting Treatment Effect Estimates by Post-Stratification in Randomized Experiments” (opens in a new tab), Luke W. Miratrix, Jasjeet S. Sekhon and Bin Yu, Journal of the Royal Statistical Society: Series B, Oxford University Press.
14. “Covariate Adjustment in Stratified Experiments” (opens in a new tab), Max Cytrynbaum, Quantitative Economics, The Econometric Society.
15. “Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data” (opens in a new tab), Alex Deng, Ya Xu, Ron Kohavi and Toby Walker, WSDM 2013, Association for Computing Machinery.
16. “CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials” (opens in a new tab), Kenneth F. Schulz, Douglas G. Altman, David Moher and the CONSORT Group, The BMJ, BMJ.