Question 314:

Asked on August 5, 2008

Tags: A/B Testing , 2-proportion test

1

Answer:

No answer provided yet.

So what you are doing is comparing two proportions and there is indeed a more "scientific" way of determining if there is a significant difference. If your sample is sufficiently large (~ above 100) you can use the normal distribution to make inferences about the difference you observe. What you want to know is the difference you see between two pages greater than what would be expected by chance alone.

I'll walk through an example. Lets assume you observe 50/200 in one version, then 30/300 in another version. The two proportions you are calculating are then .25 and .10 and the difference is .15. Given your sample size of 500 (200 in one page and 300 in another) can we conclude the difference of .15 is greater than chance?

You divide this difference by the square-root of a denominator which accounts for the chance which will provide you with a z-score. The denominator is

1         1
--   +  --     * PQ
n₁       n₂

Where P = (x1 + x2)/(n1+n2)  Q is 1-P. The x's are just the number of conversions and the n's are the sample size.

P = (50+30)/(200+300) = .16
Q = 1-.16 = .84
PQ = .84*.16 = .1344

1/n1 + 1/n2 = 1/200 + 1/300 = .0083

So multiply .1344 * .0083 = 0.00112

Now the square root of this is SQRT(0.00112) = .03346

So the equation is the observed difference .15/.03346 = 4.482

That result is the z-score which is your test-statistic. You now look this value up using the z-score to percentile calculator using the 2-sided area. You should get 0.0007 or there is less than a .07% chance the difference is due to chance.  With this data I'd conclude with a lot of confidence that the difference is statistically significant. The next question is: is the difference of 15 percentage conversion points good enough? The answer depends on what your goals are, but that sounds pretty good to me.