Question 631:

Asked on February 19, 2009

Tags: A/B Testing , Response Rates

1

Answer:

No answer provided yet.

So there are multiple parts to your question. I've worked with response rates and other "A/B testing "scenarios with other clients so I'm very familiar with this request. First, you are correct that there is indeed a mathematical way to understand the variability in your response rate estimate.  Additionally, I think you understand that regardless of how large your sample size is, there will always be a certain margin of error in your estimate. Just like with all those presidential polls we are used to hearing about (e.g 52% +/- 3%). The plus and minus figure (+/- 3%) is the margin of error and it's largely a function of the sample size.  So to answer your first question, the number of marketing events largely depends on how narrow of a margin of error you need. A margin of error of +/- 5% will require fewer events than a margin of error of +/- 1%.  

 

While it depends a bit on your sample sizes and proportions you are seeing, in general the best thing to do is start calculating confidence intervals around your proportions and you can use the Confidence Interval Calculator http://www.measuringusability.com/wald.htm For example, lets suppose you have 500 out of 9000 respond to a direct mail piece. You have an observed 5.56% response rate. The margin of error would be +/- .47% or we can be 95% confident the true response rate is between 5.08% and 6.03%.

 

The next part of your question likely has multiple components too, but let me take a crack at it. Typically marketers want to know if a conversion rate is statistically higher than another conversion rate. To test this we can use a statistical test called the 2-proportion test.  Let's say you want to know if there is a difference in response rates between two response vehicles. Using the example above and lets call that the response rate for post-cards (500 out of 9000 responded) and lets say the same ad responses for a Newspaper Insert are 350 out of 8250 for a response rate of 4.24%.  Well, is this response rate statistically different than the 5.56%?  To find out if the difference is greater than what we'd expect due to chance alone we use the 2-proportion test.  The observed difference between proportions is .013 or 1.3% and the probability this difference is due to chance alone is less than 1% or p < .0001. The formula uses the normal approximation to the binomial.

 

See also question 515 for detailed calculations and formula for the 2-proportion test and confidence intervals.