Spotify strives for team autonomy and independence. This means that no team should be blocked by others and they should be able to move as fast as they can. The autonomy has is a challenge for managing a centralised and coordinated experimentation infrastructure and analysis. This a talk about how we approach A/B testing in a fast moving company.

1. February 12, 2015
A/B testing at Spotify
Ali Sarrafi & Evan Shrubsole

2. Who is this?
2
Ali Sarrafi Evan Shrubsole
Software EngineerProduct Owner

3. Section name
Source:
Agenda
3
Spotify intro
How Spotify works
Our challenges
How we manage A/B tests
How we implement A/B tests
Conclusions + Q&A

8. February 12, 2015
The right music for every moment
8

9. 9
15 million subscribers
60 million monthly active users
30 million songs

10. Tribes
Provide fast and
reliable access to all
the world's music
Enable high product
development speed
while maintaining a
highly available service
P P P P
Squad Squad Squad Squad
Chapter
Chapter
P P P P
Squad Squad Squad Squad
Chapter
Chapter
Our team is growing rapidly
400
30
100
10

11. 11
Autonomous fast moving teams
Feature 1
Feature 3
Feature 2
Feature 4
Feature 5

12. February 12, 2015
More Context
Over 50 autonomous teams working on features
Over 7 platforms with multiple features
Over 3000 source repositories

13. Section name 13
How to make sure we improve the product?

14. 14
Hippo Approach
Highest Paid
Person’s Opinion

15. Do A/B testing!
15
A/B testing is a simple
randomised experiment
with at least two variants,
which are the control and
treatment in the controlled
experiment

16. Spotify Example
16

17. 17
We don’t want a Frankenstein software
Challenge

18. 18
Well at least a good looking 2014 edition

19. Section name 19
How can we align teams’ optimisation to
avoid the Frankenstein effect?

20. 20
Align on metrics

21. Source:
21
DAU (Daily Active Users)
MAU (Monthly Active Users)

22. Section name 22
Evolving Metrics

23. Focus on functionality as a metric
We want people to play music and enjoy it
23

24. 24
DAU/MAU
Function 1 Function 2 Function 3 Function 4
CTR/Team Metrics
Importance
(High to low)
Order of
optimisation

25. Section name 25
How should we implement this efficiently?

26. Code
26
if (shuffle_button = “green_square")
show_green_square_shuffle()
else
show_old_shuffle()

27. Managing tests
27
A/B test
Name: My test
Hypothesis: …
….
….
Registry

28. Analysing results
28
Exposure logs

29. 29
Making magic happen with desired metrics
Vital Sign Group A Group B
Functionality 1 +1.5% -0.5%
Functionality 2 +2.5% -3%
Functionality 3 +0.5% 0%
Functionality 4 +1% -7%

30. Section name 30
How can we make the process more
successful?

31. 31
It’s all about automation

32. 32
and of course education

33. Copyright
Napster pic by bixentro, cc 2.0
http://www.flickr.com/photos/bixentro/
Busy record store pic by gomattolson
http://www.flickr.com/photos/gomattolson
Empty record store pic by passetti
http://www.flickr.com/photos/passetti/
First Frankenstein screenshot form :
“Frankenstein” movie by Kenneth Branagh, 1994
Second Frankenstein screenshot form :
“I, Frankenstein” movie by Stuart Beattie, 2014
Automation screenshot from:
Startrek Next generation, TV show (1987–94)

34. Thanks