At Booking.com we have experienced what an evidence based organisation means for creating business impact. And what looks it like, when experimentation is part of your culture. This has helped Booking.com run more than 1000 experiments at the same time… each day of the week. In thi about how the world’s largest accommodations provider leverages online experimentation to scale their onboarding strategy. And how this helps Booking.com empower people to successfully host their property on the Booking.com platform.

1. Experimentation @Booking.com
Jorden Lentze
20th of September, 2018 Many thanks to Willem Isbrucker

2. 1. Booking.com, nice to meet you
2. The business impact
3. Experimentation @Booking.com

3. Jorden Lentze
Product Owner, Booking.com
● 1 year at Booking
● Previous at Google and ABN AMRO
● Conversion Rate Optimization
● 300+ experiments
● Rotterdam
https://www.linkedin.com/in/customerdrivenoptimization/

4. Timeline.
1996
Founded in Amsterdam

5. Winterberg

6. Winterberg

7. 2012
4,000 employees
2016
13,000+ employees
2000
8 employees
OVER
100
NATIONALITIES
2018
17,000+ employees

8. 28 million
total reported listings
43
Languages supported
200+
Offices
OVER
100
NATIONALITIES
161,000,000+
Reviews from real guests
1,550,000+
Room nights booked
every 24 hours

9. Agenda.
Source: Kleiner Perkins 2018 Internet trends: https://www.statista.com/statistics/277483/market-value-of-the-largest-internet-companies-worldwide/

10. Why?

11. 9/10 experiments fail.
90% of product decisions have an
inconclusive or negative effect on
a product’s primary metric
Source: VWO 2016.
(8.75/10 tests fails according to AppSumo)
90%
10%

12. Evercore Equity Research
https://goo.gl/rFP
“Booking’s utilization of A/B testing has
contributed to Booking enjoying
conversion rates 2-3x higher than the
industry average.”

13. Product Death Cycle.

16. Experimentation at Booking.com

17. Base.
Which one performed better?
Variant.

18. Hypothesize &
Experiment.
• Hypothesize
• Calculate power
• Run for prescribed time
• Stop, rerun or Full On
• Iterate
Base.
Variant.
Search
Search

19. Experiment on everything.

20. Experiment atomically.

21. Interactions.
Experiment on
Everything
Test Atomically
A lot of
experiments!
+ =

22. 1,000+
concurrent tests.

23. Based on [prior] we believe that
changing [condition]
for users [sample] will make
them [outcome].
We will know this when we see
[effect(s)] happen to [metric(s)].
This will be good for customers,
partners and our business
because [motivation].
Idea ≠ Hypothesis.
Hypothesis Template.

24. Idea ≠ Hypothesis.
• All secondary metrics
selected a priori along with
their direction of change to
avoid Texas Sharpshooter
Fallacy

25. ● All experiments should have
run time calculated
beforehand using the power
calculator of your choice.
Or use ours :-)
https://bookingcom.github.io/
powercalculator/
● Experiments should then be
run for the next closest full
week cycle.
Properly Powered.
Full Week Cycles.
Experiments should:

26. No HIPPOs.

27. Teams made for
experimentation.

28. Allow 100% access to
data.

29. Guidelines, not rules.

30. Embrace failure.

31. Fail fast.

32. Don’t assume
correlation.

33. Don’t assume
reproducibility.

34. Winterberg

35. Go where the experiment takes you

36. Thank you.
https://blog.booking.com/

37. Thanks to
Willem Isbrucker
Senior Product Owner, Booking.com
● Search products at
Booking.com
● 2 years at Booking
● Many more in Product
● 1000+ experiments
www.linkedin.com/in/willemisbrucker