How to Prove Which Change Drove Performance

You prove which change drove performance by writing a falsifiable prediction before the change ships, fixing the measurement window before you see any data, isolating the change from everything else moving in that window, then grading the result against that specific prediction, improved, hurt, or neutral. The prediction has to be on record first. That's the method. Everything else is technique for protecting the read. Skip the prediction and the best claim you can honestly make is "the account moved," which describes an outcome without identifying a cause.

Here's the part that trips up good media buyers. The hard step isn't the measurement, it's that the default state of an ad account is unprovable, and no amount of analysis after the fact fixes it. If you shipped nine changes in a two-week span with no predictions written down, there is no report, no attribution model, and no incrementality study that recovers which one worked. The evidence wasn't destroyed. It was never created. Provability is built into the change at ship time or you don't have it.

Why is the default state of an ad account unprovable?

Three reasons stack, and each alone is enough.

You shipped more than one thing. A normal week includes bid adjustments, a budget shift, two paused ad groups, a new creative, and a negative keyword sweep. All land inside the same performance window. The number afterward is the sum of all of it, plus everything you didn't do.

You never stated what you expected. Without a prior claim, any outcome is consistent with any story. ROAS up: the bid change worked. ROAS down: it worked but seasonality masked it. Both readings survive contact with the data, which means the data isn't testing anything.

The platform measured it on its own clock. Each platform reports conversions on its own attribution settings, timezone, and modeling. Google Ads' default click window is 30 days. Meta's default is 7-day click. Change something on both and read on day 10, and the Google number is still filling in while the Meta number is substantially settled. That's not a discrepancy to reconcile, it's two measurements of different things.

There's a fourth nobody likes to name: the platform is also changing your account. Smart Bidding adjusts continuously, Performance Max reallocates on its own logic, Advantage+ shifts placements. You're never the only actor, and the platform's actions don't appear in your change list.

What does a falsifiable hypothesis actually look like?

It names the change, predicts a directional effect on a specific metric, sets a threshold, and bounds the time. If you can't imagine the result that would prove it wrong, it isn't one. Here's the same intended change written both ways.

That last row is the point. The bad version can never fail. Volume up 2%? It helped. Volume flat? It would have dropped without us. The good version fails at 8%, and it fails even though performance improved, which feels wrong the first few times and is exactly what makes the record worth anything later.

The two-sided rule. A hypothesis with only an upside condition is half a hypothesis. "Conversions will rise" ignores what you were willing to give up. Real changes trade one thing for another, volume for efficiency, reach for CPA. Name the thing you're spending and the floor you won't cross. That ROAS floor is doing more work than the volume target.

When a hypothesis is the wrong tool. Not everything needs one. Fixing a broken conversion tag, removing a disapproved ad, pausing a campaign promoting a sold-out SKU, these are corrections, not experiments. There's no prediction to make because there's no alternative outcome you'd accept. Writing hypotheses for housekeeping is theater, and teams that force it abandon the whole discipline within a month. Reserve it for changes where you're genuinely uncertain.

How do you isolate a change enough to read it?

Isolation is three separate problems. Most teams solve the first and ignore the other two.

Timing

One change per campaign per window, or accept that you can't read either. Keep a shipping calendar per campaign, not per account. Two changes in different campaigns on the same day are usually fine. Two changes in the same campaign on the same day are one change with a mystery inside it.

The learning-phase caveat is real and routinely ignored. Bid strategy and budget changes push campaigns back into a learning state on most platforms, and performance during that period isn't representative of the steady state. Start your window at the moment of the change and you're reading the disruption, not the effect. Start it after delivery stabilizes, and write down when you decided it stabilized, because otherwise that decision drifts to wherever the data looks best.

Scope

Record what the change touched: account, campaign, ad group, and how much spend sits underneath it. A 15% bid change on a campaign running $200/day and the same change on one running $9,000/day aren't comparable evidence, and a verdict list treating them as equal rows will mislead you about which habits are working. Scope also sets window length, low-volume campaigns need longer to produce a readable signal, so the tests you can run fastest are the ones you can least trust fastest.

Everything else that was moving

The step that gets skipped. Before grading, write down what else was happening: a competitor promo, a site outage, a seasonal spike, a landing page change the web team shipped. You're not going to control for these. You're noting them, so a verdict rendered during Black Friday is annotated as such and doesn't get quoted six months later as clean evidence.

What are the honest limits when you can't run a clean holdout?

A true holdout, a matched, randomized group that doesn't get the change, is the only design that establishes causation outright. Usually you can't run one. Geo splits need enough volume per cell to separate signal from noise. Campaign-level splits change delivery in ways that break the comparison. And the biggest budget decisions are the ones you're least willing to withhold from half your market.

So be direct about what a hypothesis-and-verdict method is: structured, dated, falsifiable observation. Much better than reading a dashboard. Not a randomized experiment, and claiming otherwise is the overreach that gets a whole measurement practice dismissed the first time someone technical looks at it.

What it does give you: a prediction on record before the outcome, so the result can contradict you; a consistent definition of "worked" applied the same way to changes you liked and changes you didn't; a per-change record instead of a per-month one.

Weakest on single, large, one-time changes in a volatile period. One budget doubling during a seasonal peak produces a verdict that's mostly a measurement of the season. Grade it, annotate it, don't build strategy on it.

Strongest on repeated changes of the same type across many campaigns. No single verdict is conclusive. Thirty verdicts on the same change type, graded consistently, tell you something a holdout on one campaign never would, you're reading a pattern rather than an instance.

Why isn't "it improved after we changed it" proof?

Because "after" and "because of" are different claims, and the account supplies a steady stream of coincidences.

Run it honestly. Performance improved in the window. Candidates: your change, a competitor cutting spend, the platform's bidding model finding better inventory, seasonality, a change someone else shipped, or a tracking artifact where delayed conversions from the previous period landed inside your window and inflated it. That last one is a common mechanical trap on long attribution windows, the lift you're reading may belong to traffic from before the change existed.

The correlation trap is asymmetric, which is its real cost. You'll investigate changes that appear to have hurt. The ones that appear to have helped go into the weekly report and never get audited. Over a year that builds a playbook of tactics nobody tested, worse than no playbook, because it's held with confidence.

A stated hypothesis doesn't remove confounders. It forces you to commit to a threshold before you know the answer, which removes the largest source of bias: you, deciding after the fact what counts as success.

How is a graded verdict different from reading a dashboard?

A dashboard reports the state of the account. A verdict evaluates a specific prior claim. Different objects, and one can't substitute for the other.

The last row matters most. A dashboard's history is a line. A verdict history is a queryable list: every tCPA increase you've made, graded the same way, so you can see that your bid changes land improved more often than your budget reallocations do. You can't extract that from a performance chart at any resolution.

What a verdict is not: the platform's reported result for the period. That figure includes everything else in the window plus modeled conversions. Handing it back as a verdict answers a different question, the mechanics are in Your ROAS Is Lying: Cross-Platform Numbers.

And "neutral" has to be real. A meaningful share of changes genuinely don't move anything past the noise floor. If your verdicts are overwhelmingly "improved," the threshold is too loose or it's being set after the fact.

What happens when two changes overlap anyway?

They will. A client asks for something urgent mid-window, or a teammate ships into the same campaign without checking.

Two honest options. Stagger, delay the second change until the first window closes, which costs real time you sometimes can't afford. Or void the verdict, mark both changes unreadable for that window and say so. Not "improved, low confidence." Unreadable.

The tempting third option is partial credit, and it's worse than either. A framework that quietly picks a winner when the evidence can't support one produces the same confident output shape whether or not the read was clean. Once a few of those enter your track record, none of it is worth much.

Voiding feels like waste. It isn't. Sixty clean verdicts and fifteen explicit voids beats seventy-five verdicts of unknown quality, because you can act on the first one.

Does this survive at real change volume?

Manually, past a point, no, and pretending otherwise is how the discipline dies quietly.

Here's an illustrative model; run it with your own numbers. Say you're on six accounts across three platforms, shipping roughly 30 changes a month. Writing a real hypothesis: 4 minutes. Setting the window and noting scope: 2 minutes. Returning weeks later to grade it: 8 minutes. About 14 minutes per change, so roughly 7 hours a month, split awkwardly across two calendar periods because grading happens weeks after writing.

Seven hours isn't the problem. The split timing is. Grading is unattached to any deadline, invisible to anyone if skipped, and lands during whatever the following week turns out to be. So it erodes, and non-randomly, because the verdicts you most want to skip are the ones where the change you advocated for didn't work. The track record degrades in exactly the direction that makes it useless.

That's the honest case for making the hypothesis and verdict part of the change mechanism rather than a discipline layered on top. Adlyse ships each proposed change with the hypothesis already attached, grades it after execution, and automatically reverts what grades as hurt, so the record exists as a byproduct of shipping rather than as homework.

Who this isn't for: if you run one account with a handful of changes a month, a shared doc with four columns, change, prediction, window close date, verdict, genuinely works. Build the habit there first. The habit is the valuable part; tooling only matters once volume makes it impossible to sustain.

FAQ

Related reading

• Incrementality Testing: The Only Measurement That Answers the Question You're Actually Asking, the harder, cleaner version of this same question when a holdout is actually possible.