Contenu connexe Similaire à Experimenting in the Enterprise Agile 2013 Conference Similaire à Experimenting in the Enterprise Agile 2013 Conference (20) Experimenting in the Enterprise Agile 2013 Conference2. 2© 2012 BigVisible Solutions
David J Bland
Startups consulted: 18
Twitter followers: 2179
Iterations of Coaching: 215
Speaker Vanity Metrics
Brian Bozzuto
Blogs & Articles Written: 77
Presentations Delivered: 46
Teams coached: 54
3. © 2013 BigVisible Solutions 3
This session doesn’t
guarantee a 100 million
dollar success, but it should
help prevent a 100 million
dollar failure.
4. © 2012 BigVisible Solutions 4
A business model used to be static
and survived many years.
38
13
4
3
2
0 10 20 30 40
Radio
TV
Internet
iPod
Facebook
Years to Reach a Market of 50
Million
6. © 2012 BigVisible Solutions 6
The Lean Startup provides a
scientific approach to creating
and managing startups and get
a desired product to
customers’ hands faster.
-Eric Ries
9. © 2013 BigVisible Solutions 9
This is great in
theory, but it’s not for
B2B.
This is great in
theory, but I’m a
visionary.
EntrepreneurIntrapreneur
10. © 2013 BigVisible Solutions 10
Define the value proposition for:
Source: Adapted from Steve Blank
Buyer
User
Decider
Buyer
User
Decider
B2C B2B
11. © 2013 BigVisible Solutions 11
Are your users/customers:
Source: Adapted from Steve Blank
Aware of problem?
Actively seeking a solution?
Hacking together a solution?
Have $$$ for your solution?
14. © 2013 BigVisible Solutions 14
Source: Adapted from Cynefin
Simple
“Everyone Knows That”
Complicated
“Let’s consult the expert”
Complex
“Whoa, it all makes sense now…”
Chaos
“That still doesn’t make sense”
Work with experts to
inform our hypothesis
Run experiments to gain
new knowledge
17. © 2013 BigVisible Solutions 17
If you experiment just
to see what
happens, then you’ll
succeed at seeing
what happens.
19. © 2013 BigVisible Solutions 19
Image Source: Eric Ries The Lean Startup
Then here
Start here
Last here
20. © 2013 BigVisible Solutions 20
Image Source: Eric Ries The Lean Startup
Now
Execute
Quickly
21. © 2012 BigVisible Solutions 21
Experiment Template
For <customer segment>
I predict <outcome>
when I run <experiment>
22. © 2012 BigVisible Solutions 22
Experiment Example
For new customers,
I predict 10% increase in
conversion
when I offer a “no questions
asked” cancellation policy
23. © 2012 BigVisible Solutions 23
Experiment Guidelines
• Customer Segment (who)
• Up to / Limit Of (how many)
• Length of Time (how long)
• Financial Exposure (how much)
• Amplification & Dampening Measures
24. © 2013 BigVisible Solutions 24
Experiment Dashboard
ID Hypothesis Description Status Next Steps
01a
We predict
that adding
a refund
feature will
increase
conversions
Story 13c adds
refund label
and call to
action button
that puts
customer in
contact with
call center
agent.
Complete
0.2%
increase in
conversion
on cohort.
Meeting
with VP of
Ops to
discuss
strategy.
25. © 2013 BigVisible Solutions 25
“This is all great, but how
do we know what
experiments to run?”
26. © 2012 BigVisible Solutions 26
Iterative Product
+ Static Business Model
Failed Organization
28. © 2012 BigVisible Solutions 28
? ? ???
?
?
?
?
?
?
?
“Let’s map out our existing business model”
Source: Adapted from Alex Osterwalder
tip: tell a story
29. © 2012 BigVisible Solutions 29
? ? ???
?
?
?
?
?
?
?
“Let’s validate our riskiest assumptions”
Source: Adapted from Alex Osterwalder
30. © 2012 BigVisible Solutions 30
This is most
likely your
Riskiest
Assumption
Source: Adapted from Alex Osterwalder
31. © 2012 BigVisible Solutions 31
This is most
likely your
other
Riskiest
Assumption
Source: Adapted from Alex Osterwalder
32. © 2012 BigVisible Solutions 32
What happens when we
subtract costs from revenue?
Source: Adapted from Alex Osterwalder
33. © 2013 BigVisible Solutions 33
“Okay, now we have a big
list of assumptions, how do
we decide where to start?”
34. © 2013 BigVisible Solutions 34
experiment
experiment
experiment
experiment
experiment
experiment
experiment
“Let’s dot vote!”
(you could, but that’s pretty subjective)
35. © 2013 BigVisible Solutions 35
time
learning
experiment
experiment
experiment
experiment
experiment
experiment
experiment
experiment
experiment
experiment
experiment
experiment
experiment
(ok better, but what are they tied to?)
36. © 2013 BigVisible Solutions 36
Now we have a focusing mechanism
focustime time
learninglearning
experiments experiments
experiments experiments
37. © 2013 BigVisible Solutions 37
And assumptions that need validation
focus
experiments experiments
experiments experiments
time time
learninglearning
38. © 2013 BigVisible Solutions 38
This is Experiment Mapping
focus
experiments experiments
experiments experiments
time time
learninglearning
39. © 2013 BigVisible Solutions 39
Customer Segment is our focus
focus
experiments experiments
experiments experiments
time time
learninglearning
Customer
Segment
40. © 2013 BigVisible Solutions 40
Will they pay for our product?
focus
experiments experiments
experiments experiments
time time
learninglearning
Customer
Segment
They will
pay for our
product
41. © 2013 BigVisible Solutions 41
Can we scale to meet demand?
focus
experiments experiments
experiments experiments
time time
learninglearning
Customer
Segment
They will
pay for our
product
We can scale
to meet
demand
42. © 2013 BigVisible Solutions 42
Can we use our distribution channels?
focus
experiments experiments
experiments experiments
time time
learninglearning
Customer
Segment
They will
pay for our
product
We can use
existing
distribution
channels
We can scale
to meet
demand
43. © 2013 BigVisible Solutions 43
Are we solving a real need?
focus
experiments experiments
experiments experiments
time time
learninglearning
Customer
Segment
They will
pay for our
product
Our product
solves a need
they have
We can use
existing
distribution
channels
We can scale
to meet
demand
44. © 2013 BigVisible Solutions 44
Which of these is the riskiest?
focus
experiments experiments
experiments experiments
time time
learninglearning
Customer
Segment
They will
pay for our
product
Our product
solves a need
they have
We can use
existing
distribution
channels
We can scale
to meet
demand
45. © 2013 BigVisible Solutions 45
Can we do a concierge test?
focus
experiments experiments
experiments experiments
time time
learninglearning
Customer
Segment
They will
pay for our
product
Our product
solves a need
they have
We can use
existing
distribution
channels
We can scale
to meet
demand
Concierge
test with 10
customers
46. © 2013 BigVisible Solutions 46
We could build a prototype
focus
experiments experiments
experiments experiments
time time
learninglearning
Customer
Segment
They will
pay for our
product
Our product
solves a need
they have
We can use
existing
distribution
channels
We can scale
to meet
demand
Concierge
test with 10
customers
Build a
prototype
47. © 2013 BigVisible Solutions 47
Maybe run a focus group?
focus
experiments experiments
experiments experiments
time time
learninglearning
Customer
Segment
They will
pay for our
product
Our product
solves a need
they have
We can use
existing
distribution
channels
We can scale
to meet
demand
Concierge
test with 10
customers
Build a
prototype
Run a focus
group with
80 people
48. © 2013 BigVisible Solutions 48
Prototype will take time
focus
experiments experiments
experiments experiments
time time
learninglearning
Customer
Segment
They will
pay for our
product
Our product
solves a need
they have
We can use
existing
distribution
channels
We can scale
to meet
demand
Concierge
test with 10
customers
Build a
prototype
Run a focus
group with
80 people
49. © 2013 BigVisible Solutions 49
Focus groups take a long time too
focus
experiments experiments
experiments experiments
time time
learninglearning
Customer
Segment
They will
pay for our
product
Our product
solves a need
they have
We can use
existing
distribution
channels
We can scale
to meet
demand
Concierge
test with 10
customers
Build a
prototype
Run a focus
group with
80 people
50. © 2013 BigVisible Solutions 50
Concierge test is quick & we’ll learn
focus
experiments experiments
experiments experiments
time time
learninglearning
Customer
Segment
They will
pay for our
product
Our product
solves a need
they have
We can use
existing
distribution
channels
We can scale
to meet
demand
Concierge
test with 10
customers
Build a
prototype
Run a focus
group with
80 people
51. © 2013 BigVisible Solutions 51
“Let’s swarm on this
experiment”
high
learning
short
timeframe
52. © 2013 BigVisible Solutions 52
Just because you can
measure it, does not
mean you should
measure it.
53. © 2013 BigVisible Solutions 53
Auditable
Actionable
Accessible
Source: Eric Ries The Lean Startup
54. © 2013 BigVisible Solutions 54
Auditable
“Be able to defend your metrics
with confidence”
56. © 2013 BigVisible Solutions 56
Actionable
“Does this metric incentivize a
change in behavior?”
57. © 2013 BigVisible Solutions 57
Actionable?
Facebook likes won’t
influence your dog
58. © 2013 BigVisible Solutions 58
Actionable
Number of steps
should influence
your lifestyle
59. 0
50
100
150
200
250
1 2 3 4 5 6 7 8 9 10 11
Velocity(Points)
Iteration
Team Velocity per Iteration
Delta Planned Actual Avg Last 3
© 2013 BigVisible Solutions 59
Actionable?
What does this tell
you about your
product?
60. © 2013 BigVisible Solutions 60
Accessible
“Can you easily access and share
this data?”
63. © 2013 BigVisible Solutions 63
Source: Dave McClure Pirate Metrics
Acquisition
Activation
Retention
Referral
Revenue
64. © 2013 BigVisible Solutions 64
Source: Dave McClure Pirate Metrics
Acquisition
Activation
Retention
Referral
Revenue
65. © 2013 BigVisible Solutions 65
“Metrics are people, too”
Source: Eric Ries The Lean Startup
66. © 2013 BigVisible Solutions 66
Aware
Hopeful
Satisfied
Passionate
Source: Adapted from Cooper & Vlaskovits
67. © 2013 BigVisible Solutions 67
Aware
Hopeful
Satisfied
Passionate
Acquisition
Activation
Retention
Referral
69. © 2012 BigVisible Solutions 69
When you have stories
around an uncertain
feature, add a hypothesis
that states the expected
outcome on a specific
metric.
70. © 2012 BigVisible Solutions 70
If you have a Kanban Board,
add a Validate Column to
the very right of it.
71. © 2012 BigVisible Solutions 71
Problem Solution Process
Known Known
Waterfall or
Agile
Known Unknown Agile
Unknown Unknown Lean Startup
72. © 2012 BigVisible Solutions 72
Where are the
unknown / unknown’s
in your enterprise?
Notes de l'éditeur Overall, pedometer users increased their physical activity by 26.9% over baseline. An important predictor of increased physical activity was having a step goal such as 10,000 steps per day (P = .001). When data from all studies were combined, pedometer users significantly decreased their body mass index by 0.38 (95% CI, 0.05-0.72; P = .03)." Promoting Lifestyle Physical Activity: Experiences with the First Step Program.Am J LifestyleMed. 2009 Jul 1;3(1 Suppl):508-548. Overall, pedometer users increased their physical activity by 26.9% over baseline. An important predictor of increased physical activity was having a step goal such as 10,000 steps per day (P = .001). When data from all studies were combined, pedometer users significantly decreased their body mass index by 0.38 (95% CI, 0.05-0.72; P = .03)." Promoting Lifestyle Physical Activity: Experiences with the First Step Program.Am J LifestyleMed. 2009 Jul 1;3(1 Suppl):508-548.