Cast Study
Learning quickly through experimentation
One small change delivered a 4.38% uplift in booking conversion
We ran 1000's of experiments across the customer journey. Each one was designed to answer a question quickly, learn from real customer behaviour and give us evidence for what to test next.
Designing and leading experimentation at scale, test everything.
Rather than commit time to large redesigns, we tested one idea at a time, measured the effect and used each result to decide what to do next.
That process helped us move quickly, reduce risk and build a much stronger understanding of the drivers behind conversion.
I worked across UX, UI and CRO, helping identify opportunities, form hypotheses, design experiments and use the results to determine what we tested next.
My Contributions
UI & UX design
Experiment design
CRO
Analytics
Strategy
Team
Product
Design
Engineering
Analytics
Approach
Small changes
Fast implementation
Large test audiences
Evidence led iteration
Start with an assumption
“The countdown timer in the checkout drives conversion.”
The booking timer was a legacy feature and there was a strong internal belief that it encouraged customers to complete their booking.
When the ASA required us to change its messaging as the offer wasn’t strictly held, rather than simply making it compliant, we used the opportunity to test whether that assumption was actually true.
Then I’d make the experiments themselves the biggest part of the page.
Our testing philosophy
How it works
When working on the pre-purchase user experience of the site, the primary objective is to improve overall conversion as quickly as possible, i.e.make the largest improvements in conversion rate for the greatest number of visitors.
To do this effectively we use experiments to try to learn what makes visitors on our site complete a purchase, what factors influence their decisions and behaviour.
We called these ‘Drivers of Purchase Intent’. These drivers aim to capture precisely the underlying why of any changes seen in purchase behaviour.
It is important to understand that these drivers were not limited to what a visitor would self-declare as things they care about, like or need. They include anything that results in a greater likelihood of making a purchase.
There are three main types of drivers:
- Functional – the tools we give users to be able to search, select and pay for a holiday
- Emotional – messaging and framing that affect users’ perception of what they are seeing and their decision-making
- Design – the way we present features that make them easier to use
What are drivers?
Every page and component on the site can be assessed in terms of what drivers & what fundamental need to a purchase decision it’s addressing.
Which driver works best?
There is limited space on a page and in a user's capacity to focus and therefore continuous trade-offs must be made between which drivers to address and in what manner in any given location.
The aims of drivers
The aim of running experiments is to develop a very deep knowledge of exactly the best ways to address the most valuable drivers in an given location on the site.
Small, small, big, big
Small effort, small number of variables, big enough change for customers to notice, big enough audience to make a change.
Test one thing without weeks of development
See whether customers behaviour actually changes
Understand why it changed
Turn the result into the next hypothesis
Keep compounding what we learn
List of identified drivers
- Price - Promotion
- Price - Framing
- Price - Value
- Price - Competitiveness
- Price - Affordability
- Urgency - High Demand
- Urgency - Low Supply
- Urgency - Time Constraint
- Urgency - Deal Window
- Focus & Clarity - Demotion
- Focus & Clarity - Explanation
- Focus & Clarity - Prominence
- Focus & Clarity - Rephrasing
- Commitment - Time
- Commitment - Input
- Commitment - Reciprocity
- Merchandising - Hotels
- Merchandising - Flights
- Merchandising - Destination
- Merchandising - Extras
- Merchandising - Suitability
- Trust - Credibility
- Trust - Fulfilment
- Trust - Security - Payments
- Trust - Security - PII (Personally identifiable information)
- Excitement - Travel Experience
- Excitement - Deal Winning
- Excitement - Desirability
- Excitement - Rarity
- Assurance - Social Validation
- Assurance - Reassurance
- Assurance - Simplicity
- Usability - Choice Reduction
- Usability - Hurdle Removal
- Usability - Interaction Improvement
- Usability - Personalisation
- Usability - Performance
How it works in practice
Challenging a legacy assumption
The booking timer (a countdown for how long user had complete the checkout process) was considered untouchable. There was a long standing belief that it was a major conversion driver. This belief was based largely on anecdotal knowledge rather than current evidence, and the assumption was so strong that changing or removing it was considered too risky. When we were forced to remove the timer messaging, we had an opportunity to test that belief properly.
The result: removing it made no measurable difference to conversion.
That changed the conversation completely. Instead of accepting the legacy assumption, we now had evidence that the timer itself was not doing the job everyone thought it was. If urgency can influence booking behaviour, how should we design it so customers actually notice and respond to it?
Test 01
What happens if we remove the timer?
We removed the timer completely to test whether it really influenced conversion.
This challenged a long standing internal belief that the timer was a major driver of bookings.
Hypothesis: Removing the timer will reduce bookings.
Result
No measurable difference
Removing the timer had no measurable impact on conversion.
What we learned
The timer itself wasn’t doing the job everyone thought it was.
Next question
If urgency can influence booking behaviour, how should we design it so customers actually notice it?
What we learned
The timer itself wasn’t doing the job everyone thought it was.
Test 02
Reintroduce a more noticeable timer with compliant messaging
With the original assumption challenged, we reintroduced the timer using ASA compliant messaging.
This allowed us to start testing urgency itself, rather than relying on the legacy version of the timer.
Hypothesis: Clearer messaging could make the purpose of the timer easier to understand.
Result
+4.38%uplift in booking conversion
What we learned
Making the urgency message clearer produced a measurable uplift. This gave us evidence that urgency could influence booking behaviour.
Next question
If urgency works, could increasing its visibility strengthen the effect?
What we learned
Making the urgency message clearer produced a measurable uplift. This gave us evidence that urgency could influence booking behaviour.
Test 03
Make the timer more prominent
We increased the visibility of the timer and made the consequence of leaving the booking clearer.
The aim was to test the impact of the feature when customers actually saw it, rather than simply testing different visual treatments.
Hypothesis: If customers notice and understand the timer, urgency will influence more bookings.
Result
+4.02%uplift in booking conversion
What we learned
Urgency can influence purchase behaviour when customers actually notice it.
Next question
Would creating even more urgency by reducing the time increase the impact further?
What we learned
Urgency can influence purchase behaviour when customers actually notice it.
Test 04
Test the limits
We reduced the timer from 30 minutes to 15 minutes to understand whether increasing the time pressure would drive conversion further.
Hypothesis: If the timer is creating urgency, reducing the available time from 30 minutes to 15 minutes should increase the pressure to act and drive more bookings.
Result
No change
Reducing the available time did not improve conversion.
What we learnt
More urgency wasn’t automatically better.
Next steps
Focus on how urgency is communicated and understood, rather than simply increasing the pressure on customers.
What we learned
More urgency wasn’t automatically better.
Test 05
Can fear of missing out drive conversion?
Testing the timer duration suggested that time pressure itself wasn’t driving the change in behaviour.
Our hypothesis shifted. Customers appeared to be responding to the possibility that they might miss the offer, rather than to the countdown itself.
So we kept the timer and looked for other simple ways to reinforce that same driver.
What changed
We added a small “Don’t miss out” message alongside the price, complementing the existing timer.
It was deliberately a very small change. Before investing in more sophisticated ideas, we wanted to understand whether simply reinforcing the risk of missing the offer could influence behaviour.
Hypothesis: Reinforcing the possibility of missing the offer will encourage more customers to act and move towards booking.
Result
+2.03%
uplift in package booking conversion
97% statistical significance
3.3m+ visitors exposed to the experiment
What we learnt
Fear of missing out was influencing customer behaviour.
The timer duration hadn’t mattered, but reinforcing the possibility of losing the offer did.
And importantly, we achieved this with an extremely small change: two words placed alongside the price.
This was “small, small, big, big” in action. A tiny change, quick to implement and focused on a single idea, produced a 2.03% uplift in package booking conversion across millions of customers.
The impact
Small changes at massive scale
At loveholidays, millions of customers meant even small improvements could have a significant commercial impact. Our challenge was finding those improvements quickly and understanding why they worked.
What started as a question about an “untouchable” timer became a much deeper understanding of what influenced customers to book. By testing small changes quickly, we challenged assumptions, separated time pressure from fear of missing out, and used each result to decide what to test next.
The value wasn't just the individual conversion uplifts. It was creating a repeatable way to learn quickly, reduce risk and turn customer behaviour into better product decisions.
Test small. Learn fast. Scale what works.