Start Free
Free Tool

A/B Test Calculator

Plan an experiment or analyze its results, sample size, statistical significance, and confidence intervals in one place. No signup, no email gate, runs entirely in your browser.

Your current conversion rate for the metric you're testing.
Smallest relative lift you want to be able to detect (e.g. 5% means 10% → 10.5%).
Probability of detecting the effect if it's real. 80% is standard.
Acceptable false-positive rate. 5% is standard, two-sided.
For more than two arms we apply a Bonferroni correction to α.
Used to estimate how long the experiment will run.
Sample size per variant -
Total visitors needed -
Estimated test duration -

-

How this calculator works

Sample size (Plan mode)

We use the standard two-proportion sample size formula for a two-sided test with equal allocation between control and variant:

n = ( z1-α/2 · √(2 · p̄ · (1 - p̄))
    + z1-β   · √(p1(1-p1) + p2(1-p2)) )2
    / (p2 - p1)2

Where p1 is the baseline rate, p2 = p1 · (1 + MDE), and p̄ is the mean of the two. With more than two variants we apply a Bonferroni correction by dividing α by the number of comparisons.

Significance (Analyze mode)

We run a two-proportion z-test with pooled variance for the test statistic, then build the confidence interval on the absolute difference using the unpooled estimator:

z = (p2 - p1) / √(p̄(1-p̄)(1/n1 + 1/n2))
p-value = 2 · (1 - Φ(|z|))
CIdiff = (p2 - p1) ± zc · √(p1(1-p1)/n1 + p2(1-p2)/n2)

The relative-lift CI is the absolute-difference CI divided by the control rate. "Probability B > A" is computed from the same z statistic and is a frequentist analog, not a Bayesian posterior.

Frequently asked questions

What is a minimum detectable effect (MDE)?

The smallest lift the experiment can reliably detect at the chosen power and significance level. A 5% MDE on a 10% baseline means you'll be able to detect a true rate of 10.5% or higher; smaller real effects will frequently be missed. Smaller MDEs require dramatically more traffic, halving the MDE roughly quadruples the sample size.

Why is "peeking" at results bad?

Standard p-values assume you commit to a sample size in advance and look once. Checking results repeatedly inflates the false-positive rate well beyond the nominal α. If you need to monitor an experiment continuously, use sequential or always-valid p-values, or commit to a full-sample readout.

Should I use one-sided or two-sided tests?

Default to two-sided. A one-sided test only protects against false positives in one direction, which is rarely what you want, a "neutral" change that meaningfully hurts a metric is usually still bad news.

What about Bayesian A/B testing?

Bayesian methods (posterior probability that B beats A, expected loss) are a great alternative when you want to reason about uncertainty rather than a binary verdict. We're adding a Bayesian mode soon, bookmark this page.

Is this calculator suitable for revenue or other continuous metrics?

Not yet. v1 only handles binary conversion outcomes (proportions). For continuous metrics like average revenue per user or session length, you'd want a t-test or a bootstrap, which we're planning to add.

Common mistakes this calculator can't catch
  • Sample ratio mismatch (SRM): a meaningful imbalance between control and variant traffic usually points to an instrumentation bug, not a real effect.
  • Novelty effects: users react to anything new. Run experiments long enough for the novelty to fade.
  • Multiple comparisons: testing many metrics or segments without correction inflates the chance of a false positive somewhere.