Question 162:

Asked on April 22, 2008

Tags: Confidence Intervals , Control Chart , A/B Testing , p-chart

1

Answer:

No answer provided yet.

If your historical conversion rate is 11% and you want to see a 2% improvement ( I believe you mean a 2 percentage point improvement ) or a 13% conversion rate one strategy is to generate a 95% confidence interval around the observed conversion rate. If the lower end of the interval is above 13%, then you've reached your goal. If you're splitting your traffic between two web-page versions, your daily sample size will be about 140 on each per day. So using that as the sample, you'd need to see at least 27 conversions out of 140 visitors to be 95% confident the conversion rate is at least 13%. I got this result from using a two-sided confidence interval around a proportion.

There are many variables and assumptions of course. For one, we need to assume that one day is representative of your typical traffic and conversions. Second, the normal fluctuations in traffic mean that just because you see 27 out of 140 does not mean your ongoing conversation rate is 13%. On the other hand, if you don't observe that in one day, it also doesn't mean the conversation rate is below 13%. What you'll likely need to do to establish an ongoing conversation rate is use a control chart (probably a p-chart) to establish the typical upper and lower limits of the conversation rate for your control and new web-version.