I run many experiments in my role and knowing how to read the data, and deciding what to do next is paramount to success. In this blog I’m looking at a simple funnel experiment for a user flow that’s not converting as well as I would like and from digging in to the data I think it’s due to users not finding sessions to book.
The hypothesis for this experiment is
Users who see more available sessions book more.
Now an easy answer here is to say don’t show unavailable products, but this particular product is date dependent, like booking a beauty treatment. Depending on the day you would like to book depends if we have an appointment and or therapist free that day.
My control journey for this is experiment is customers who follow this:
- Listing page and choose a treatment
- Go to the product page and proceed to the basket
- In the basket they are asked for a date
- If the date is available they can proceed to the checkout and pay
- If the date is unavailable they have to go back to the listing page and start again
As I’m not sure of the best solution here between two new journeys we develop the two new user journeys.
Journey 1. Forces the user to tell us their prefered date or dates and shows all the sessions available that day.
Journey 2. Allows the user to go to the product page, but offers them to check availability and see all sessions available that day before adding to the basket.
So we have three journeys:
1. (Blue): Control – Product page only – Current Journey
2. (Green): Force the user to give us a date and show all sessions / beauty therapists available on that day before going to the product page – Journey 1
3. (Teal): User can choose to check availability before the basket – Journey 2
Below are the initial results
Experiment results
Only 1.44% of users in the control group become customers.
When we force users to select a date we increase conversion to payment to 1.69%.
When we allow users to choose to check availability we see a clear winner at 4.65% conversion. Or do we? With a jump that big my gut doesn’t trust it, and if my gut is wrong I have just struck a gold mine, so I dig deeper and look at the numbers closer.
Experiment Digging
Total number of users that went through each journey:
Journey 1: 834 users
Journey 2: 827 users
Journey 3: 43 users
As you’ll see, the third version (teal), where we allowed the user to choose to check dates only 43 people choose to do that (almost 5% of the control group), you could say this is a false positive and the real winner here is journey 2 (green), where we force the user to select a date.
My first question here is did we prove the hypothesis? Users who see more available sessions book more
Yes we did, in both the experiments we increased sales.
Next experiment steps
The second question now is, how can I get more people to check the availability and book a session? As journey 2 was only just better, and I want significant change. Note this experiment needs many more bookings to show significance.
So I write a new hypothesis for a new experiment around the third (teal) journey – product page before checking availability.
Users who see the product before checking availability book more
This means we need to start a new experiment!
The new control will be to Force the user to give us a date and show all sessions / beauty therapists available on that day before going to the product page
My new experiment will be to force the user to check availability from the product page. This could be a very simple experiment, but I also have another question percolating in my mind.
What should be the experience when the user finds the product unavailable? I certainly don’t want a dead end. Here I want to give the user alternative options, of other sessions or other dates
Find out how that works out in my next blog (coming soon, after the experiment has matured).
All data used in this blog is example data only for demonstration purposes.
Related
Startups are full of metrics and numbers, especially when they start to look for investment. Ask any investor in tech and I would bet you that every one of them has come up against a company which initially looks like they have great metrics, but when they dig deeper they…
Much like what I expect it's like to train your dragon (if you're crazy enough to own one), roadmaps can be difficult to train, they can be big, they can be hard to handle, and they can also be very... mythical too. Also, like works of fiction they can just…