Statistical significance in the CRO context is the measure of confidence that the observed performance difference between two tested variants (for example, variant A versus variant B) is real and not the result of chance or natural data variation.
The standard confidence level used in testing is usually 95%, meaning there is only a 5% probability the observed result is random – declaring a winner before reaching this significance level is a common error leading to decisions based on insufficient data.
Why it matters
Prematurely stopping an A/B test based on early results that look promising but haven’t reached statistical significance is one of the most common sources of flawed CRO decisions, which can lead to implementing changes with no real benefit.