A/B test calculator: is it a winner, or just noise?
Check whether a test result is statistically significant, see the range the real lift probably sits in, plan how many visitors and weeks a test needs before you launch it, and catch a broken traffic split before it fools you.
An A/B test calculator compares the conversion rates of two versions of a page and tells you how likely it is that the difference between them is real rather than random noise. This one uses a two-proportion z-test, the standard method for conversion rates, and reports the p-value, the relative uplift and the range the true uplift probably sits in. It also sizes tests before launch, including tests with more than one challenger, and checks the traffic split for sample ratio mismatch. Everything runs in your browser.
Nothing you type is stored or sent anywhere. The calculator runs in your browser. It uses a two-proportion z-test for significance, the standard two-proportion formula for sample size, with a Bonferroni correction when there is more than one challenger, and a chi-square test for the traffic split.
How the A/B test calculator works
Conversion rate is conversions divided by visitors, worked out for each version. The calculator then asks a narrow question: if the two versions really converted at the same rate, how often would random chance alone produce a gap at least this large? That probability is the p-value.
To get it, the calculator pools both versions into one conversion rate, works out the standard error of the difference between them, and divides the observed difference by that error. The result is a z-score, and the p-value is read from the normal distribution. A p-value of 0.03 means a gap this large would turn up about 3 times in 100 if the change did nothing. When the p-value falls below your threshold, 0.05 at 95 percent confidence, the result is statistically significant.
The uplift range shows where the true lift probably sits at your chosen confidence level. Read it next to the p-value. With the calculator’s default numbers the result is significant, yet the range runs from about a 2 percent lift to a 38 percent lift, and those two ends make very different business cases.
Two things a p-value does not tell you: the probability that B is better than A, and how big the lift will be after rollout. It only tells you how surprising the data would be if there were no difference at all.
How to read your result
- Significant, B ahead. B is likely better. Look at the low end of the uplift range: if it barely clears zero, the real gain may be small. Then check that the test reached its planned sample size, ran for full weeks and did not hurt guardrail metrics such as average order value or subscription take rate.
- Significant, B behind. The change likely hurt conversion. That is a useful result: it tells you something about what your shoppers value, and it stopped you shipping a loss.
- Not significant. The data cannot separate the two versions yet. If the test has not reached its planned sample, keep it running. If it has, call it inconclusive. An inconclusive test is not a loser, and it is not a quiet winner either.
- Uneven split. If visitors were not divided the way you planned, run the sample ratio check before you read anything else.
How long should an A/B test run?
Decide before launch, not while watching the graph. The sample a test needs depends on three inputs: the page’s current conversion rate, the smallest lift worth detecting (the minimum detectable effect, or MDE), and how sure you want to be. Lower conversion rates and smaller lifts need far more visitors. Halving the MDE roughly quadruples the sample.
The calculator uses 80 percent statistical power by default: if the true lift equals your MDE, the test will detect it about 8 times in 10. Divide the total sample by daily visitors to the tested page to get the duration, then round up to whole weeks so weekday and weekend shoppers are both in the result.
| Current conversion rate | Visitors per version to detect a 5% lift | To detect a 10% lift | To detect a 20% lift |
|---|---|---|---|
| 1% | 637,000 | 163,100 | 42,700 |
| 2% | 315,200 | 80,700 | 21,100 |
| 3% | 207,900 | 53,200 | 13,900 |
| 5% | 122,100 | 31,200 | 8,200 |
| 10% | 57,800 | 14,800 | 3,800 |
Lifts are relative, so a 10% lift on a 2% conversion rate means 2.0% to 2.2%. Figures assume 95 percent confidence, 80 percent power, a two-sided test and one challenger against the control.
Read across the 2 percent row. Detecting a 5 percent lift takes about 315,000 visitors per version; a 20 percent lift takes about 21,000. That is why lower-traffic stores get more from bold changes, such as a new offer or a rebuilt product page, than from button colors. If a test would need more than about eight weeks, change the plan: test a bigger idea, run it on a higher-traffic template, or use fewer versions.
Six ways A/B tests produce winners that are not real
- Peeking. Checking a test every day and stopping the first time it crosses 95 percent pushes the real false positive rate far above 5 percent, because random swings cross the line early and often (Evan Miller). Set the sample size first and read the result once it is reached, or use a tool built for sequential testing.
- Too little data. Early in a test, a handful of orders can swing the conversion rate by double digits. A result read far short of its planned sample is mostly noise.
- Too many comparisons. Test five versions, or check ten metrics, and one of them will look significant by chance. The sample size tab corrects for extra versions. For metrics, choose one primary metric before launch.
- Sample ratio mismatch. If a 50/50 test ends up 52/48 on large traffic, something is dropping or misassigning visitors, and the conversion result cannot be trusted until you find out what.
- Partial weeks and promotions. A test that runs Monday to Thursday, or straddles a sale, measures that stretch of traffic, not your normal shoppers.
- Winning on the wrong metric. A version can lift conversion rate while cutting average order value or margin. Judge winners on revenue per visitor or profit as well as conversion rate.
The calculator is the arithmetic. A testing program is the rest.
Significance math is the easy part. The results that move revenue come from choosing what to test from real research, writing the hypothesis and primary metric before launch, sizing every test, QA on every device, and keeping a record of what losing tests taught you. That is how we run A/B testing for DTC brands, as part of The CRO Program. For the basics in more depth, read our step-by-step guide to A/B testing.
A/B test calculator FAQs
01What is statistical significance in an A/B test? +
A result is statistically significant when the difference between the versions is larger than random variation would usually produce. At 95 percent confidence that means a p-value below 0.05: if the change did nothing, a gap this large would appear less than 5 times in 100.
02What confidence level should I use for an A/B test? +
95 percent is the usual default. Use 90 percent for low-risk changes that are easy to reverse when traffic is tight, and 99 percent for changes that are expensive to roll out or undo, such as pricing. Choose it before the test starts, not after you see the result.
03How many visitors do I need for an A/B test? +
It depends on your current conversion rate and the smallest lift you want to detect. At a 2 percent conversion rate, detecting a 10 percent relative lift at 95 percent confidence and 80 percent power takes about 80,700 visitors per version. Use the sample size tab above for your own numbers.
04Can I stop an A/B test as soon as it reaches significance? +
Not if you have been checking it along the way. Stopping at the first significant reading inflates false positives, because random swings cross the threshold early and often. Decide the sample size in advance and read the result when it is reached, or use a sequential testing method designed for continuous monitoring.
05What is a sample ratio mismatch? +
A gap between the traffic split you planned and the split you got that is too large to be chance. It usually points to a technical problem, such as a redirect that loses visitors, a version that breaks on some browsers or bots filtered unevenly. Fix it before you trust the conversion result.
06Should I use a one-sided or two-sided test? +
A two-sided test asks whether B is different from A in either direction. A one-sided test only asks whether B is better. One-sided tests reach significance with less data but cannot tell you that a version made things worse. Use two-sided unless you decided otherwise before launch.
07Is this calculator frequentist or Bayesian? +
Frequentist. It reports a p-value and a confidence range from a two-proportion z-test. Bayesian tools report a probability that B beats A instead. On a properly sized test the two usually point to the same decision. With either method, the bigger risk is stopping early.
More free tools and next steps
Conversion rate calculator
Work out your conversion rate and what a lift is worth in revenue per year.
ROAS calculator
Return on ad spend, break-even ROAS from your real margin, and the target CPA you need.
A/B testing services
Testing run as a program, with sized tests, written hypotheses and a weekly scorecard.
The CRO Program
Research, design, development, QA and analysis as one monthly program, judged on profit.
Landing pages for paid traffic
One page per ad angle, built and tested by the team that runs the tests.
Want tests that reach a real answer?
Send us your site and your monthly traffic. We will show you which pages have enough traffic to test, what is worth testing first and how long each test needs, free and with no obligation.