When I visited Israel I have also participated on one front-end event in Booking.com Tel Aviv office. This was very inspiring event with examples from Booking.com teams. I took these five basic lessons away.
1. The goal in product development is not being right but to be progressively less wrong
This is clear and well said. Most experiments that happen in Booking.com are not eventually set for all users and after testing they are stopped. Anyway, it makes great sense to continue with experiments as you never know what will be the result.
2. Probability vs. Distribution
For example if you take average salary 20k and you have 10 people. This result may be achieved with following figures:
- 1, 1, 1, 1, 2, 2, 4, 4, 50, 134 or
- 5, 10, 10, 15, 15, 15, 20, 25, 30, 70
This array of numbers may apply on salaries of footballers and programmers. Meanwhile in football, the probability of having great salary is really low, in programming you usually have a good salary, but you probably never achieve the salary of Cristiano Ronaldo. The average will be the same.
You should never focus on average user. We should focus on median user if we want to approach the most costumers from the target group.
3. When testing – never underestimate compounding variable
According to the researchers, more people die on the boat having a lifejacket than without it. Does it make sense to wear it?
Yes, because there is one compounding variable – weather. People usually wear the lifejacket only if the weather is bad. During every experiment, Booking.com evaluates vast number of compounding variables – from weather to important news in the TV.
4. Sunk cost fallacy
When the first Concorde airplane left the company, everybody had already known that it can never be in profit. But, the investments were huge. Or if you start watching some TV series – event if it is a bullshit after 3rd season, you watch it anyway. If you invest too much in some feature, sometimes is difficult to stop it – you should.
5. Technology will not give you high performance!
Booking.com still uses old-fashioned technology and it is still the biggest and growing site for hotel bookings? Why? Because, users don’t care. These are all technologies that Booking ignored and it is still rocking:
I have also some smaller notes:
- 80:20 this is the optimal way of being lazy
- Facebook has exact know-how how many people user must connect after sign-up to stay on Facebook
- Interesting think is also Survivorship bias (recommend Wikipedia article)
- If launching experiment, Booking.com measures the people that have seen the experiment separately
- In Booking.com developers has all the data to know the areas to focus on
- Booking uses jQuery for testing – quick and easy to set-up
Simply, great session with inspiring speakers. You can also read my report from Israel (in Czech).
