What is A/B significance?
Significance is the confidence that the difference between variants is not due to chance. Convention is 95% confidence, a p-value below 0.05.
How the calculator works.
This is a two-sided two-proportion z-test. It pools the rates under the null, computes the standard error, and returns the p-value from the normal distribution.
A worked example.
Control 200/5,000 = 4.00%. Variant 240/5,000 = 4.80%. Lift = 20%. The z-test returns a p-value around 0.07, just shy of 95% significance: a real-looking lift, not yet confirmed.
How to use it well.
Decide your sample size first with the sample size calculator. Run to that number. Do not peek and call it early; that inflates false positives.
What it does not tell you.
Whether the effect persists, whether it matters financially, or whether you measured the right metric. Statistical significance is a guardrail, not a recommendation.