KPI’s: Painting a car red won’t make it go faster

Using the correct metrics

When it comes to conversion rate optimization, defining and monitoring the correct metrics (KPI) for your AB test is the first thing you should be doing.

In other words, not every single AB test you run, should have order rate* or AOV* as the primary KPI. That’s not to say there won’t be a residual impact, but the sale may take place much further down the funnel that you don’t even notice it. However, you should learn the value of the metrics you will be measuring. This way, you can truly understand the benefit of optimising towards them and the impact they will have on the bottom line.

A correct understanding will prevent incorrect interpretations either by yourself or someone else, allow you to build a case to take your optimisation to greater places and make it easier to tell a story of how conversion was improved.

*(AOV, sales, revenue, orders, etc. – I call these macro KPI’s. These are KPI’s that your company will be measured against but can’t just improve by themselves. You need to pull levers elsewhere to impact these metrics, these levers are your micro KPI’s. I will talk about macro vs. micro KPI’s in another post)

Value of these metrics

Understanding the value of your metrics is an important concept. You can’t just will your conversion rate to improve, you have to do something that will later result in a sale. Sometimes the impact will be much later on and other times it will be just a step or two away.

Below is an example of non-macro KPI’s you can measure along with companies that might want to measure them.

Registrations (SkyBet), adding to basket (ASOS), viewing products (Amazon), watching a video (Lynda), creating a product (Nike), downloading some information (Ford), making a phone call (Vitality insurance), signing in (Netflix), building a pizza (Papa John’s Pizza).

There are countless more metrics that you probably didn’t even know you could measure. The key is to go out and find them and not assume that they have to be your companies financial goals, however you do need to know the value they will add if they improve.

Case Study (fake data)

Let’s look at a hypothetical Papa John’s example.

PJ’s decide to run an AB test on customisation of their pizzas. They know that customers who customise their pizza are more likely to add to basket than the customers who don’t. Once in basket, customers roughly checkout 50% of the time.

The hypothesis – “Customers who customise their toppings are more likely to add to basket”.

The KPI’s – 

• Incorrect: Conversion Rate primary
• Correct: Add to basket primary and conversion rate secondary.

The current user journey –

Select an “any pizza any size for £11.99” offer.

Clicking “Choose a Pizza” takes you to the following page:

The test – the variation will take customers straight to the customisation page once they’ve selected their deal and the control will remain as choosing a pre-built pizza, which can be customised once they’ve selected their deal.

The results (fake):  

• 1st tier = traffic
• 2nd tier = added to basket
• 3rd tier = order placed

Interpretation – Below are a few interpretations of the results above, of which only one is correct.

1. The p-value for the overall conversion rate (10% and 12%) is 0.125 meaning the results are strong in favour of the variation but not conclusive. NO WINNER YET
2. The overall conversion rate is better by 2% points in favour of the variation. VARIATION WINNER (no consideration to statistical significance)
3. Basket to order rate is lower for the variation  (P-value 0.247 / statistical significance not considered): CONTROL WINNER
4. Add to basket rate is 6% points higher in favour of the variation group, P-value is 0.005 (primary KPI). No statistically significant impact on secondary KPI. VARIATION WINNER, although you could leave the test running to see if your secondary KPI has improved.

 Conclusion – 

From the analysis above you can see that:

• Point 1 is technically correct but if you look at the primary KPI you can conclude a winner.
• Point 2 is also technically correct, but it is by complete fluke that the interpretation drawn happens to also be the right conclusion. This is where not knowing the value of your metrics can land you in trouble.
• Point 3 is incorrect as it is analysing the part of the experience which was unchanged. No changes were made to basket, hence the basket to order rate didn’t have a significant change.
• Point 4 is correct as it looks at the metric that was impacted by the test and reaches a conclusion faster than the macro KPI, which would possibly still need a few more weeks to reach the same conclusion.

You can’t monitor speed if all you’ve done is change the colour of the car!