Check if one ad really beats the other, or if the gap is still chance.
Rate = conversions ÷ visitorsp = all conversions ÷ all visitorsz = (rate B − rate A) ÷ √(p × (1 − p) × (1 ÷ visitors A + 1 ÷ visitors B))p-value = 2 × (1 − Φ(|z|)), significant at 95% when under 0.05Visitors per ad = (1.96 × √(2p̄(1 − p̄)) + 0.84 × √(p₁(1 − p₁) + p₂(1 − p₂)))² ÷ (p₂ − p₁)², with p₁ the baseline rate, p₂ = p₁ × (1 + lift), p̄ = (p₁ + p₂) ÷ 2A gap this big would show up less than 5% of the time if both ads performed the same. Strong evidence, not proof.
Yes. Put impressions in visitors and clicks in conversions.
The test needs at least 5 expected conversions and 5 expected non-conversions per ad. Below that its math does not hold.
When each ad has reached the sample size and the result is significant. Stopping the first time it looks significant picks winners that are not real.