KDD 2016

1. Making fashion recommendations with
human-in-the-loop machine learning
Brad Klingenberg, Stitch Fix
brad@stitchfix.com
Machine learning meets fashion
KDD2016 | San Francisco | August 2016
Three lessons

2. Personal styling recommendations with humans in the loop
Fashion with humans in the loop:
It works really well, but it’s complicated

3. Personal styling recommendations with humans in the loop
Fashion with humans in the loop:
It works really well, but it’s complicated
Lesson 1: You have more than one feedback loop
Lesson 2: Human selection changes your objective function
Lesson 3: Even humans need feature selection

4. Humans in the loop for fashion
at Stitch Fix

5. Stitch Fix

6. Stitch Fix

7. Stitch Fix

8. Stitch Fix

9. Styling at Stitch Fix
Personal styling
Inventory

10. Styling at Stitch Fix: personalized recommendations
Inventory
Algorithmic
recommendations
Machine learning

11. Styling at Stitch Fix: expert human curation
Human curation
Algorithmic
recommendations

12. Lesson 0: Humans and machines
are a winning combination

13. Combining art & science
Humans and machines are a winning combination.

14. Combining art & science
Humans and machines are a winning combination.
Human judgement
● helps leverage unstructured data

15. Combining art & science
Humans and machines are a winning combination.
Human judgement
● helps leverage unstructured data
● provides a human element to the
process (empathy, creativity)

16. Combining art & science
Humans and machines are a winning combination.
Human judgement
● helps leverage unstructured data
● provides a human element to the
process (empathy, creativity)
● frees the algorithm developer from
edge cases

17. Lesson 1: You have more than
one feedback loop

18. Traditional recommenders
Learning through feedback

19. Humans in the loop
Learning through feedback

20. Humans in the loop
Learning through feedback
Human interaction

21. Lesson 2: Human selection changes your
objective function

22. Training a model
What should you predict?

23. Training a model
What should you predict?
Historical shipment data is plentiful

24. Training a model
What should you predict?
Naive approach: ignore selection and train on historical shipment data
Advantages
● “traditional” supervised problem
● simple historical data

25. Training a model
Problem 1: selection can censor your data

26. Censoring through selection
Problem: selection can censor your data

27. Censoring through selection
Problem: selection can censor your data

28. Censoring through selection
Problem: selection can censor your data
Arms flaunted
Success
Yes
No
Yes No
?
?
p
1-p

29. Success given selection
Problem 2: success probabilities can make
for terrible recommendations

30. Success given selection
Problem: success probabilities can make for terrible recommendations

31. Success given selection
Problem: success probabilities can make for terrible recommendations
“Probability the
stylist should
choose this item”

32. Success given selection
Problem: success probabilities can make for terrible recommendations
“Probability the
stylist should
choose this item”
Probability the stylist will be able to
send this to the right client

33. Success given selection
clients
Y_i = 1
Y_i = 0
Example: A low-coverage item

34. Success given selection
clients
Y_i = 1
Y_i = 0
An edgy dress. Not many people will like it, but
easy for a stylist to identify these clients
Example: A low-coverage item

35. Success given selection
clients
Y_i = 1
Y_i = 0
High score!
Example: A low-coverage item

36. Success given selection
Example: A high-coverage item
clients
Y_i = 1
Y_i = 0

37. Success given selection
clients
Y_i = 1
Y_i = 0
More neutral item. Many clients will like it, but
it is hard for stylists to identify these clients
Example: A high-coverage item

38. Success given selection
clients
Y_i = 1
Y_i = 0
Average score!
Example: A high-coverage item

39. A useful recommendation?
High score
Low score

40. The lesson
Effective recommendations require
understanding human selection
In both of these cases,

41. Effective recommendations require understanding human selection
Generally, selection data will be much larger and more complicated to
collect and work with
● Negative cases: logging the set of things that was available to
be selected but was not selected
● Presentation effects
○ similar to the way search engine results are studied

42. Lesson 3: Even humans need feature selection

43. Recall the styling process
Human curation
Algorithmic
recommendations

44. Creating features for the human classifier
Unstructured data can be overwhelming, even for humans

45. Creating features for the human classifier
Derived feature 1
Derived feature 2
Feature engineering: creating useful summaries for human consumption

46. Creating features for the human classifier
Derived feature 1
Derived feature 2

47. Creating features for the human classifier
It is important to focus on
● Interpretability
● Evidence
● Orthogonality
For fashion data especially, this is hard!

48. Creating features for the human classifier
A / B
Ultimately, this is an empirical question
Run experiments with
● Production systems
● Simulations for human classifiers

49. Personal styling recommendations with humans in the loop
Fashion with humans in the loop:
It works really well, but it’s complicated
Lesson 1: You have more than one feedback loop
Lesson 2: Human selection changes your objective function
Lesson 3: Even humans need feature selection

50. Thanks!
Questions?