Incrementality Holdout Sample Size Calculator
This calculator determines the minimum holdout group size required to detect a statistically meaningful incremental lift from your advertising, given your audience size, baseline conversion rate, and the minimum lift you expect to measure. It is built for media planners, growth marketers, and data scientists designing geo-based or audience-based incrementality tests who need to set holdout parameters with statistical rigor. The tool supports both user-level holdouts and geo-level holdout designs.
Inputs
Results
How this is calculated
Enter your total addressable audience size, baseline conversion rate, minimum detectable lift, and desired confidence level, and the calculator returns the required holdout percentage, holdout group size, and expected test duration.
Every model runs locally in your browser. Nothing you type is sent anywhere.
Frequently asked questions
What is an incrementality holdout test?
An incrementality holdout test measures how much of your observed campaign performance is caused by the advertising itself rather than by consumers who would have converted anyway without seeing the ad. A randomly selected holdout group is withheld from campaign exposure, and the conversion rate of the exposed group is compared to the holdout. The difference between the two rates is the incremental effect of the advertising, also called the true causal lift.
How large does my holdout group need to be?
Holdout group size depends on your baseline conversion rate and the minimum lift you want to detect. Lower baseline rates and smaller expected lifts require larger holdouts to achieve statistical power. As a rough guideline, detecting a 10% incremental lift on a 2% baseline conversion rate typically requires a holdout of at least 20,000 to 50,000 users, depending on your confidence threshold. This calculator gives you the precise figure for your inputs.
What is the difference between a holdout test and an A/B test?
An A/B test typically measures the relative performance of two ad creatives, landing pages, or audience segments. An incrementality holdout test measures whether advertising as a whole is causing conversions that would not have happened without it. Holdout tests answer the question of true causal impact; A/B tests answer the question of relative efficiency between options. Both are necessary for a complete measurement program.