The Importance Of Centralising Experiments

AB Testing Programme Management For Free

Before I begin, let me clarify two things. 1) this post is not sponsored and 2) the tool (Illuminate) really is 100% free.

I am writing this post for no reason other than the fact that Illuminate helped us solve a genuine problem (for free). A problem I’m sure many organisations face. I also want to be clear that this post is about the problem and not the tool itself.

The Problem

Stop me if this sounds familiar. Countless teams running their own experiments. An equal number of roadmaps stored on Google Sheets, each with a different format. Valuable knowledge of historical tests sitting on personal drives or emails. Time wasted trying to track down the outcomes of past tests only to hit a brick wall. Experiment records lost due to team members moving on. Test integrity compromised as a result of conflicting tests. In general, no clear view of what’s been, what is and what will be running.

Sounds horrible right? How can you ever claim to have a successful culture of experimentation if you don’t have a handle on the experimentation programme? How can you tell it’s growing if you can’t quantify it?

No central knowledge base — This was the challenge my team and I faced.

What about Google Sheets, JIRA or Trello I hear you ask? They’re not fit for purpose and you’ll see why throughout the rest of this post.

The Solution

After researching a few tools I came across Illuminate by Brooks Bell. I created an account and began playing around with it. Honestly, I couldn’t believe the tool was free.

What is Illuminate? Illuminate is a program management tool that allows you to centralise and store all your experimentation knowledge into one place. This includes completed, live and upcoming tests, roadmaps, test ideas, insights we’ve learned, reports, win rates and so much more. There are some limitations, such as integrations with other platforms, but honestly, simple is better than complex.

7 Features That Helped Us Solve The Problem

1. Overview of our experimentation programme (image 1)
Any company looking to build a culture of experimentation needs to know how their programme is doing so an easy-to-view dashboard was a big win for us. Features I love on the dashboard — win rate %, impact on commercial metrics, volume of tests launched, graph of completed tests. You also have the ability to look at an overview of each team, brand or channel.

Why it’s important: This allows a company/team to track the progress of their experimentation programme and its maturity level over time. It’s also a great way to monitor OKRs.

Suggestions for improvements: Give users the ability to change the graph to other metrics such as “experiments launched”.

2. Experiment roadmaps in one place (image 2)
If you’ve ever built roadmaps on Google Sheets you’ll know how much of a chore this is to maintain, especially when there are multiple roadmaps from many different people. Having all roadmaps in one place means no more inconsistent Google Sheets.

Why it’s important: Having a single place to store all past, current and future tests democratises the process, makes it easier to plan, prevents cross-contamination, and makes for better collaboration across teams.

Suggestions for improvements: Longer timeline view to make long term planning easier

3. Filters to find experiments (image 3)
When you’ve got a large volume of tests at different life stages, from different teams and on different parts of the website, it’s useful to be able to quickly filter down to the right set of tests. This is a critical factor in being able to centralise experiments. No one wants to scroll through everything to find a single test.

Why it’s important: When you’re in weekly experimentation meetings, you want to focus on the experiments that are relevant to that team or meeting. Having the ability to filter by test status, life stage or team means you can run effective experimentation meetings.

Suggestions for improvements: Slight lag in the interface when selecting/de-selecting filters.

4. Transition ideas into tests (image 4)
One of the most interesting parts of any AB Testing programme is coming up with hypotheses and creating a backlog of test ideas. So it helps to have those ideas stored in one place where they can be filtered and revisited during meetings. Having the ability to promote ideas into experiment tickets with one click helps to create a seamless experience between workflows.

Why it’s important: Running an ideation session is great but it’s important to document those ideas in a workflow that ensures there are no breaks in your experimentation process. Having ideas stored independently away from your test backlog is a good way to lose them!

Suggestions for improvements: None

5. Score test ideas (image 5)
Every person I’ve ever spoken to who runs an experimentation programme uses some type of prioritisation framework to score their ideas, e.g. RICE. This is why I felt that the scoring feature in Illuminate was a nice little touch on top of being able to brainstorm and store test ideas.

Why it’s important: You can’t test everything in one go! Prioritising your tests ensures that you run your most critical experiments first.

Suggestions for improvements: None

6. Customising to meet our needs
Off-the-shelf solutions often leave you feeling like you wish you had more control in tailoring the product to your organisation’s needs. I mean let’s face it, every organisation will have different needs. Illuminate allowed us to really personalise the little things like creating our own metrics, teams, devices, audiences, pages, and marketing channels that we could use to filter everything.

Why it’s important: Admittedly, this point is more about the platform but an important point nonetheless. If you are going to centralise your experimenation programme, it’s important to allow teams to tailor the programme to their needs and use language that is relevant to them.

Suggestions for improvements: Allow goals to include a metric format that includes “%’s” (or show more decimal places on the dashboard :]

7. Consistent experiment information (image 6)
We actually solved this prior to using Illuminate, but if we hadn’t this would have been the perfect solution. I’m adding this to the list for anyone who doesn’t have consistency in their experiment design/brief stage and wants to ensure each test captures important information.

Why it’s important: Capturing a certain amount of information ensures that each test is well thought through, the details of the test are logged and screenshots are captured as a reminder of the changes.

Suggestions for improvements: none

Final Thoughts

There are a lot more useful features in the tool that I haven’t touched on here. They include reports, AB Test calculators, annotation of different variants, insights and more. I feel these are more “nice to haves” and not critical in solving the problem to centralise experiments. They do help create a well-rounded tool however — for example, the Insights section is a great way to bring qualitative insights into the process. This is arguably a critical feature and one we use quite a bit, but I’ll leave it to the reader to make that call.

It’s not impossible to centralise an experimentation programme with tools like Google Sheets or Trello, but hopefully you can see how the features I’ve mentioned above really help to enrich the overall experience and make it far more engaging for everyone involved — instead of a boring JIRA board or lifeless Google Sheet!

Please do clap, comment and share if you’ve enjoyed this post. I’d love to hear your thoughts on alternative solutions.

As always, you can follow me on Twitter or with minimal effort you can follow me and CRAP Talks on Medium. Happy experimenting!

Below is a short video of the tool in action.