Incrementality Testing for Mobile UA: Geo Holdouts and PSA Tests
Incrementality testing measures the installs and revenue your ads actually caused, by comparing a group exposed to your campaign against a matched holdout that saw nothing. On mobile, the two designs that survive contact with a privacy-first stack are geo holdouts (turn the channel off in some markets, leave it on in others) and user-level lift tests like PSA and ghost ads. Last-touch attribution can't answer the causal question at all. It just tells you which pixel fired last.
That distinction stopped being academic the day ATT landed. When most of your iOS installs arrive with no device identifier, the whole edifice of "this click led to that install" gets shaky. Adjust's 2025 benchmarks put industry-wide ATT opt-in at around 35% in Q2, and that's the generous reading, counting apps that time their prompt well. Flip that number over: roughly two-thirds of iOS users hand you an install with no IDFA attached. You're modeling the majority no matter what your MMP dashboard implies.
So the honest question isn't "which campaign gets credit for this install." It's "how many of these installs would I have gotten anyway, for free?" Incrementality is the only method that answers it.
Why last-touch flatters everyone
Here's the mechanism, on-device, step by step. A user taps your Search Ads listing, installs, opens the app. Last-touch records: Search Ads gets the install. Clean, satisfying, wrong. That user was already searching your brand name. They'd have found you organically thirty seconds later. The ad didn't create demand, it intercepted it and billed you for the privilege.
Multiply that across retargeting, brand keywords, and any channel that fishes downstream of intent, and last-touch systematically over-credits the campaigns closest to the conversion. The industry consensus by 2025 is blunt about it: last-click tells a flattering story that doesn't match reality. I've watched a team cut a "top-performing" retargeting line by 40% and lose almost no installs. The attribution model had been reporting those installs as caused. They weren't.
Incrementality flips the frame from crediting touchpoints to causing outcomes. You accept that you can't see every path (you can't, not on iOS, not since April 2021) and instead run an experiment: withhold the ads from a comparable group and measure the gap.
The three test families, ranked
There are more than three variants floating around, but for mobile UA they collapse into three usable families. I've ranked them the way I actually think about them: budget cost first (that's the constraint that kills tests), then precision.
| Test type | What you withhold | Budget cost | Precision | Best when |
|---|---|---|---|---|
| Geo holdout | Whole channel, in some markets | Medium (lost installs in holdout geos) | Medium-high | You buy on channels without a native lift tool (Search Ads, DSPs, OOH-adjacent) |
| PSA test | Real ad, replaced by a placebo | High (you pay to serve unrelated ads) | High | You need a clean control and can afford placebo spend |
| Ghost ads | Nothing served; auction win is logged | Low | High | Your network or DSP supports ghost/ghost-bid logging |
A note on the user-level pair. In a PSA test, the control group gets served a real but unrelated ad (a public service announcement, historically), which lets you identify exactly which control users would have seen your ad and compare only those. Remerge's breakdown of these designs makes the trade-off plain: PSAs strip out the noise but you're paying to show people ads for something they didn't ask about, and the placebo isn't a true apples-to-apples stand-in for your brand creative.
Ghost ads fix the cost problem. Instead of serving a placebo, the system records which control users would have won your impression in the auction, then shows them nothing (or a competitor's existing ad) and tags them for comparison. Eric Seufert's writeup on Mobile Dev Memo is the canonical explainer here, and the appeal is obvious: you get PSA-grade cleanliness without the placebo-spend line item. The catch is you need a network or DSP that supports the logging. Many still don't, or gate it behind a managed-service tier.
If you're buying on a channel with no lift tooling at all, geo is your answer. Which is why the rest of this piece is mostly about designing one.
Designing a geo holdout that holds up
Geo works by splitting your footprint into two sets of markets that behave like statistical twins, running ads as normal in one set, and going dark in the other. Amsive's guide to defensible geo tests frames it well: the control markets have to be genuinely comparable, not just "the states we care about less." Do it right and the difference in installs between the two groups, over the test window, is your incremental lift. Do it wrong and you've measured a heat wave in Phoenix.
Five things to get right, roughly in order.
Match markets on behavior, not population. You want test and control geos whose historical install and revenue curves move together. The GeoLift-style literature I trust puts the bar high: matched markets with strong historical correlation, ideally north of 90-95%, and at least ten to fifteen markets in play so no single city dominates. Andava's framework walks through this and stresses that matching quality is the whole ballgame, more than test size.
Feed it enough history. You can't estimate natural variance from last month. The same GeoLift guidance calls for several months of clean baseline data (six is a common floor) before the test starts, so the model knows what "normal" divergence between your markets looks like. Skip this and you'll mistake ordinary weekly wobble for causal lift.
Size the effect before you run. Run a power analysis to find your minimum detectable effect. The MDE is the smallest lift the test can reliably catch given your market count, baseline volume, and duration. A worked example from Towards Data Science is a useful gut-check: treating around 3 cities for 21 days against roughly 10 controls has the power to detect lifts near 4-5%, and anything subtler needs more markets, more spend, or a longer window. If you're hoping to prove a 2% lift with three cities and two weeks, the math will laugh at you. Well-powered mobile tests usually land in the 2-5% MDE range, and hitting the low end costs real scale.
Run it long enough to catch latency. Install-to-value on mobile isn't instant. A user might see your ad Tuesday and install the following weekend. Tests shorter than about three weeks systematically underreport lift for anything above pure last-click intent, because you cut the window before the delayed conversions land. Two to four weeks is the usual band. Err long.
Freeze everything else. No new creative in the test markets, no price changes, no PR stunt in your control cities. The experiment assumes the only difference between the groups is the ad exposure. Every uncontrolled variable is noise dressed as signal.
Once it's running, resist the urge to peek and stop early. The math for statistical significance assumes a fixed horizon. Stopping the moment the lines diverge is how you convince yourself a coin flip is a trend.
Reading the lift without fooling yourself
At the end you'll compare installs (or purchases, or D7 revenue) in the treatment markets against the counterfactual the model builds from your controls. The gap, scaled back up, is incremental installs. Divide spend by that and you get incremental cost-per-install, which is almost always higher than the CPI your MMP reported. That's not the test being pessimistic. That's the test removing the organic installs your attribution had been quietly counting as paid.
Two failure modes to watch. First, a wide confidence interval that straddles zero means you didn't have the power to conclude anything, not that lift is zero. Report the interval, not just the point estimate. Second, one incrementality result is a snapshot of one season and one creative set. Re-run quarterly, because the number drifts as your audience saturates.
This is where a first-party analytics layer earns its keep, since the whole thing depends on trustworthy install and revenue counts split by geo. Some product-analytics platforms can define the market split, hold out a region, and read the resulting lift directly; Kixo, for instance, supports geo holdouts and reads the lift off its own event data, which saves you stitching MMP exports to a spreadsheet. Whatever you use, the requirement is the same: clean, geo-tagged, first-party events you can slice by cohort. If you're still deciding how to structure that measurement plumbing, our privacy-first attribution reference architecture sketches where incrementality sits relative to SKAN postbacks and MMP modeling.
Where geo and user-level tests fit together
You don't pick one forever. The stack most mature UA teams settled into by 2025 blends aggregate SKAN postbacks for directional in-flight optimization, MMP LTV cohorts for allocation, and incrementality (geo or lift) as the periodic ground-truth check that recalibrates everything else. SKAN tells you a campaign moved. Incrementality tells you whether the movement was real.
And yes, SKAN postbacks still arrive on their own schedule, batched and delayed by Apple's privacy timers, which is its own reason not to build your ground truth on them. (I've mostly made peace with the postback lottery. Mostly.) Incrementality doesn't wait on a postback. It waits on your test window, which you control.
If your deep-link and channel setup is a mess, the geo split will be too, because you can't cleanly withhold a channel you can't cleanly identify. Sorting that out first is worth the detour; our roundup of deep-linking platforms compared covers the routing side.
FAQ
Is incrementality testing worth it for a small app? If your monthly UA spend is low four figures, a full geo test may not have the volume to reach 80% power at a sensible MDE. Start with a single-channel on/off holdout in your biggest markets and treat the result as directional, not gospel.
Geo holdout or user-level lift test? Use geo when the channel has no native lift tool (Search Ads, most DSPs, anything offline-adjacent). Use PSA or ghost ads when the platform supports them and you want tighter per-user precision. Ghost ads if you can get them, since they skip the placebo-spend cost.
How often should I re-run? Quarterly is a reasonable default. Incrementality decays as you saturate an audience, so a result from six months ago and a fresh creative set aren't the same measurement.
Does SKAN replace the need for this? No. SKAN gives you privacy-safe aggregate outcomes, which is useful for optimization, but it's still an attribution signal, not a causal one. It can't tell you what would have happened without the ads.