This class will present hypothesis-driven development, the cutting-edge paradigm for evolving validated products. We’ll dive into how to frame hypotheses, design experiments, and use A/B testing to gather data to prove or disprove our ideas.
2. identify experiments to test hypotheses
understand how to do outcome-based planning
describe hypothesis-driven development
understand why small batches are important
define A/B testing and the culture it enables
learning outcomes
6. @jezhumbleJeff Gothelf “Better product definition with Lean UX and Design” http://bit.ly/TylT6A
hypothesis-driven delivery
We believe that
[building this feature]
[for these people]
will achieve [this outcome].
We will know we are successful when we see
[this signal from the market].
7.
8. COST OF EXPERIMENTS
8
Production
Software
SPEED
COST
new services
feasibility spike
service
substitution
integration
Quantitative
forecasting
real-time price
experiment
Data sampling
and modeling tests
Sketches &
Paper Prototypes
Interactive
Prototype
Software
demo
Interviews
& surveys
micro-niche
Wizard of Oz
VIABILITY (BUSINESS) | DESIRABILITY (CUSTOMER) | FEASIBILITY (TECH)
9. exercise
• choose a hypothesis from your assignment
• design an experiment to test your hypothesis
• what do you expect the results to be?
• what result will confirm your hypothesis?
• what result will disprove your hypothesis?
• how soon can we get the result?
10. “Etsy’s Product Development with Continuous Experimentation”
Frank Harris and Nellwyn Thomas | http://bit.ly/19Z5izI
11. “Etsy’s Product Development with Continuous Experimentation”
Frank Harris and Nellwyn Thomas | http://bit.ly/19Z5izI
12. “Etsy’s Product Development with Continuous Experimentation”
Frank Harris and Nellwyn Thomas | http://bit.ly/19Z5izI
13. Jon Jenkins, “Velocity Culture, The Unmet Challenge in Ops” 2011 | http://bit.ly/1vJo1Ya
14. do less
“Evaluating well-designed and executed
experiments that were designed to
improve a key metric, only about 1/3 were
successful at improving the key metric!”
“Online Experimentation at Microsoft” | Kohavi et al | http://stanford.io/130uW6X
15. “I think building this culture is the key to
innovation. Creativity must flow from
everywhere. Whether you are a summer intern
or the CTO, any good idea must be able to seek
an objective test, preferably a test that exposes
the idea to real customers. Everyone must be
able to experiment, learn, and iterate.”
http://glinden.blogspot.com/2006/04/early-amazon-shopping-cart.html
17. higher quality
you can stop at any time with a working system
faster feedback (assuming people pay attention)
higher motivation
less rework
working in small batches
Don Reinertsen, Principles of Product Development Flow, ch5.