In the Spirit of CRAP #14, A Contender for the Longest Blog Title Ever…

AI and Customer Empathy, Human-Centred Machine Learning, Phased BETAs and Centralising Testing (and an Astounding Opera Performance)

Arriving at the Busuu offices again for the second time in CRAP Talks history felt like I was coming home. Walking into the room I felt this calming energy that everything was under control in a familiar environment. People were gathered amongst the colourful stool chairs and talking amongst each other, pizza in hand, the quiet hum of their voices in a happy tone, excited for the night of speakers. In the beginning, Bhav addressed some of the feedback collated from the last event, the main piece revolving around length of the event overrunning. Conscious of improving this metric, Bhav let Parveen take the floor to introduce some of our magnificent speakers.

Speaker 1

Mike Migliore, Head of Customer Value at the Times, the Sunday Times and News UK delivered a charismatic presentation around putting the personal back into personalisation. He explained what his team are responsible for delivering and how the newspapers’ customer marketing and value management strategy has developed over the last decade to embrace AI.

Not only did he address the traditions of the customer experience from the birth of the editorial industry in the 1950s (painting a picture of the classic Ma n’ Pa paper shop, the owner of which would remember a customer’s preferred read) but how the Times have evolved to accommodate this digital age online for their customers without compromising on product quality or price to compete with the other mainstream spreadsheets.

He spoke about how the most successful companies put their customers at the heart of what they do, which means doing positive things for their customers. He introduced the complex intricacies which allows their digital concierge named JAMES to deliver better individual customer experience in a medium like editorial content which is perceived to be applicable to all.

Working on functional algorithms related to time, content and format of news consumption, they were able to test what is best for their customer and adapt using data driven marketing accordingly. Combining convenience with personal investment not only resulted in the reduction of churn and improvement of retention rates, but also maintained their impressive ratio regarding return on investment which is unmatched by their less expensive competitors.

Speaker 2

Ammar Jawad, an AI and Personalisation Product Manager at Hotels.com treated us to an entertaining breakdown of machine learning and its value in tracking the most peculiar of things. He began by defining what machine learning is and should be, narrowing this down to four elements. These were: acting as individuals at scale, using mass amounts of data to determine content relevance, automation thus removing the need for tagging more tedious tasks, and self-improving over time. He summarised the instances and industries for which this worked and for those in which it didn’t, and the importance of testing time frequencies depending on the situation, providing some amusing real-life examples.

They are using machine learning at Hotels.com to solve novel customer problems. He shared an example of how an airline cancelling flights from a particular location can be an automated signal for the bot to understand the context of users, anticipate their needs and map these to the machine.

He also spoke about how crucial it was to understand the evolution of a customer using their product, and how their lifestyle needs may change to the type of journey they’re looking for or in fact how many they are travelling with (business vs leisure, family additions, seasonality etc…). Considering these external events, Hotels.com have to constantly be aware of evolving with these applied filters and extract the intent of their customer habits.

What was most fascinating to learn was how Ammar championed AI as being an invaluable tool to recognise patterns surrounding criminal behaviour with the assistance of a human-in-the-loop approach. Referencing the Netflix war-on-drugs documentary series Dope about policemen monitoring drug smugglers using AI, Ammar also shared an example of how an identical sequence of supposedly random travel destinations from multiple fictionalised customers also ranked as the top countries in the world for human trafficking. Try teaching a robot that how dangerous optimising this would be!

Speaker 3

Philip Law, Head of Insight at Your Favourite Story, an analytics consultancy, spoke about a new CMS roll-out project they embarked upon with Mercedes which gave rise to the transition from the traditional “Vanilla Experience” to the spectacular “Neapolitan Experience”. In the knowledge that people hate change in general, Philip provided us with some key relatable examples to explain the reasons for their phased BETA, rather than a “Big Bang” or “Section by Section” approach to revamping websites. The sheer scale of the project was revealed through some epic statistics about the locations and retailers which were involved, as well as the manual mapping of 5,000 web pages to the new BETA site!

Sharing both the planning process, as well as the data hygiene and monitoring stages of the project, Philip was able to walk us through some of the finest tools which enabled it to be a success, including a load balancer to split to A/B testing traffic, a full website tag-checking crawler and simple score-from-100 dashboards to understand their progress.

Being honest and realistic about the difficult headaches and hiccups along the way, the team were able to innovate a new Realtime dashboard data flow tool which only took them four days to create. It was living proof that sometimes the less access you have to clean data and modern processes in the beginning can result in great and surprising achievements.

Speaker 4

Patrick Rodowicz, the Head of Insight at simplybusiness gave us a view into how they centralised testing across the company in his visionary Lab. He spoke about the fundamental elements which make it a success today in spreading the value of his team across the business, and his learning process in a constantly maturing idea.

They started by altering the smaller, independently-responsible teams into bigger teams who focused on product discovery, execution, experimentation and a build-measure-learn mentality, creating an audit trail of value.

The elemental staples of the success of this project came down to analytics (always!) in addition to organisation and collaboration. On the data side, they implemented Bayesian A/B testing frameworks, Python Notebooks and Monte Carlo simulation. Regarding organisation, they created a Trello board to track measurements on the projects which the whole C level stakeholder suite used to give them statistical confidence.

By having all the results available, increased collaboration followed, challenging beliefs in data in a positive way and making a seemingly voluntary activity something which everyone wanted to be a part of, which just goes to show they’re pretty flexible for a business insurance company.

Something Special

For the something special section, we were treated to an operatic recital from Mike Migliore, who performed the vocals for Widmung by Robert Schumann. Before moving into CRM marketing, Mike revealed that he was a Masters’ graduate in Musicology, specialising in operatic vocal performance. His amazing musical talent stunned and blew the audience away. The sheer power, depth and control of his singing voice was remarkable and it was slightly surreal to see the opening speaker return at the end for something totally special. He specifically requested that the set was not recorded, however his delivery was wonderful and the joy on his face was indescribable. You really missed out not being at this one (overall!).

As the night drew to a close, it was realised that we were actually under the time limit of our predicted schedule, which only proved that each talk was skilfully performed with exceptional pace of delivery. The content of each talk was the perfect balance between stimulating but understandable which was well suited to the audience (who I guarantee gained value from their expertise!) The core themes which emerged effortlessly throughout each unique talk topic made me feel like all the speakers were in synch with their views.  Looking forward to the videos soon!