A/B Test Significance Calculator
Tools › Testing & Measurement

A/B Test Significance Calculator

This calculator determines whether the difference between two A/B test variants is statistically significant, using a two-proportion Z-test and allowing you to set your own confidence level. It is designed for CRO practitioners, product managers, and digital marketers who run controlled experiments and need to distinguish genuine performance differences from random noise. The tool reports p-value, confidence level, and a plain-language interpretation so the result is understandable to stakeholders who are not statisticians.

live — runs in your browserOnlinePost-campaign

Inputs

Results

The frequentist test answers 'could this difference be luck?'; the Bayesian read answers 'how likely is B actually better?'. Peeking at sequential results inflates false positives — for continuous monitoring use mSPRT-style always-valid inference.

How this is calculated

Enter the number of visitors and conversions for your control and test variant, choose your confidence threshold, and the calculator runs a Z-test and returns whether the result is significant, along with the p-value and confidence interval for the difference.

z-score (pooled)
p-value (two-tailed)
significant at p < 0.05
Relative uplift
Bayesian P(B beats A)
Source: Beta-binomial posterior, Monte Carlo

Every model runs locally in your browser. Nothing you type is sent anywhere.

Frequently asked questions

What does statistical significance mean in an A/B test?

Statistical significance means that the difference in conversion rates you observed between your control and test variants is unlikely to have occurred by chance, given the sample sizes involved. At 95% confidence, you are accepting a 5% probability that the observed difference is a false positive — a real-looking result that reflects random variation rather than a genuine performance difference. Higher confidence levels reduce the false positive risk but require larger sample sizes to achieve.

What sample size do I need for a valid A/B test?

Required sample size depends on your baseline conversion rate, the minimum effect size you want to detect, and your chosen confidence level. A test designed to detect a 10% relative improvement on a 3% baseline conversion rate at 95% confidence requires roughly 12,000 visitors per variant. Smaller detectable effects and higher baseline rates require larger samples. Running a sample size calculation before starting a test is critical — stopping early because results look promising is a major source of false positives in CRO.

Can I run more than two variants at once?

Yes, but multi-variant or multivariate tests require proportionally larger sample sizes and careful correction for multiple comparisons. When testing three or more variants simultaneously, the probability of observing at least one false positive result increases with each additional variant. A Bonferroni correction or similar adjustment to your significance threshold helps control the false positive rate when running multi-variant experiments.

References

Related tools

Incrementality Holdout Sample Size CalculatorPlan a holdout or lift test in plain language: how big each group must be to detect the liiOS Signal Loss EstimatorHow much of your iOS conversion signal survives ATT — deterministic, modeled and missing.