Know Your Baseline Conversion Rate

Baseline Conversion Rate

Like with anything, if you want to improve a number, you need to have some understanding of what the baseline looks like. This could be your running time, weight or site speed. So of course, an improvement in conversion rate requires knowledge of the existing conversion rate.

It sounds obvious, but in my experience – limited such that it may be, I’ve found that conversion rate optimisation focuses on the individual test instead of the bigger picture. You often hear about annualised revenue extrapolated from the results of a single test. This is complete nonsense. A single test isn’t going to tell you if you’ve improved your website conversion. Just look at your company’s past performance. We know that eventually there is going to be a regression to the mean, we also know that the control group conversion doesn’t always reflect your site’s true conversion so using the difference between that and the variant makes no sense.

Why Test At All Then?

Showing improvements to a metric that fluctuates so much day to day is not easy to do. You’re up against external influences (or confounding variables as they are properly known) such as micro and macroeconomic factors as well as internal business decisions. Thomas Cook going out of business is a good example of this as it would have had a tsunami style effect on the entire travel industry. Brexit no doubt, is likely to be upsetting the state of how the housing market should have been behaving. So why test at all and does it really matter if you don’t know your baseline? This is a good question, I mean if things are going to change anyway who cares what the past looks like.

This is a valid argument, but you can’t guarantee that there will always be external factors, however if you know what your past looks like, you can try and predict what your future might be so that you can take steps to improve it. Most importantly, you can even quantify what impact external factors have had to your KPI’s.

But why test if you can’t extrapolate the value into future value? The primary goal of testing is not to determine the magnitude of difference between your control and variation. The goal is to simply determine that a difference exists and we do that using statistics by saying the means of our sample sets are unlikely to originate from the same population. Note my use of the word unlikely. In statistics, the only certainty is that there is no certainty, which is why we use 90%, 95%, 99% confidence and never 100%.

The Purpose of Baseline Conversion Rates

Which brings me to the point of this blog post. Knowing your baseline conversion rate is critical because if you can’t quantify the uplift of each test, you should be monitoring the overall expected conversion with the achieved conversion on a regular basis. With any “luck” you’ll see your conversion exceed your expectations with every test that is deemed a winner. And if you’re doing this frequently enough your conversion should never regress back to the old mean, instead you should start creating a new mean!

Final Thoughts

Quantifying the difference between test groups is risky and inaccurate (maybe a difference range), but

• if you can say with confidence a positive difference exists
• and you’re doing this test after test after test
• and you know your baseline conversion

you should see and be able to calculate an uplift from your test portfolio.

TL;DR – Know your baseline conversion rates!

In my next post I’ll talk about calculating your baseline conversion and forecasting. If anyone wants to read a good book on forecasting, I highly recommend Superforecasting by Philip E. Tetlock and Dan Gardner. I’ve linked to their site instead of Amazon because screw Amazon.