In the beginning stages of the our team at a former employer (large fintech company) we had a clear mandate from Execs about what to build….but the team was skeptical. We devised a crude way to validate assumptions based on the Lean Methodology. Overtime this evolved into the EDD process. Video of presentation here: https://goo.gl/Hg2c4q

1. Joshuah Vincent
@joshuahvincent
joshuahv.com
Evidence Driven DesignA lean product development methodology

2. Why Evidence Driven Design (EDD)

3. Using EDD helps ensure we have
conversations around value to build features
that have greatest relative impact.
All the stuff we
could build

4. High level Product/Feature Description
Product Description Sheet
Product
Description
Score Roadmap

5. Make decisions
with data
Use data where you have it and
identify your assumptions
Measure the return on investment
for what you are building
Run experiments to turn
assumptions into foundations
EDD VISION

6. Product
Description
Overview of the
product idea
Score
Calculating ROI
of a product
Roadmap
Activities needed
to build and ship
the product
EDD Process Overview

7. Product Description

8. Product Description Sheet
Customer
Who is this for?
For whom are we creating value?

9. Product Description
Problem
Whats the top problem (or 3) facing this
customer segment?

10. Product Description
Value Proposition
How are we uniquely solving their need?
Compelling message that states why we are
worth buying

11. Product Description
Solution
Top ways we will solve the
problems

12. Product Description
Key Business Success Metrics (KPIs):
How do we measure success?

13. Product Description
Early adopter:
How Who is the potential early
user of the solution?

14. Product Description
Alternatives
How are users solving this today?

15. Product Description
Channels
How are going to acquire
customers? Drive awareness? Keep
users?

16. Product Description
Key Stakeholders
Whose buy in do we need? Who are
key influences to stakeholders?

17. Product Description
Key Resources/Partners
Critical internal and external
resources need to deliver soln.

18. Product Description
Revenue/Business value
What’s the value to the business for
delivering the product?

19. Product Description
Cost structure
Resources/expenses for building
and maintaining product.

20. Defining Product/Feature in Terms of ROI
EDD Score
EDD Product
Description:
Overview of the
product idea.
Similar to an exec
summary.
Score
Spreadsheet for
calculating ROI of
a product
EDD Roadmap:
Includes all
activities needed
to build and ship
the product.

21. Relative indicator of ROI for a feature/product over the long term
in $/effort.
Relies on assumptions so not perfect or precise. Used as metric
to compare initiatives side by side.
Way to prioritize by generating a value that can be used to rank
features.
What is an EDD Score?

22. 6 Metrics to calc. EDD Score
Impact Per User
Customer Acquisition
Cost
Churn Rate
Cost to Serve Months to ReleaseCost to Build

23. Start Small
Calculating the EDD score
doesn’t have to take a lot of
time, invest only enough to
make the next decision.
Do the Very Simple exercise
If there is enough yield there, do the
Simple and/or Moderate exercises
Do the Detailed exercise to fully
comprehend the project compared to
other initiatives

24. Very Simple - 5 min

25. Simple - 10 minCritical Numbers
Impact per user (monthly) $13.00
Customer Acquisition Cost $0.00
Staff needed 7
Months of development before release 6
Churn rate (monthly) 2.0%
Cost to serve (monthly) $2.00
Customers in Year 3 300
EDD Score 34
7 critical numbers
Estimate when no data is available

26. Moderate - 30 min+
Consider the customer growth
Apply an attach rate for each year
to the user numbers
Will attach change over time?
Will critical numbers change over
time?
Critical Numbers
Impact per user $13.00
Customer Acquisition Cost $0.00
Churn rate (monthly) 2.00%
Cost to serve $4.00
Cost to build $697,200
Months of development before release 6
Customers
Customers in Year 1 9
Customers in Year 2 60
Customers in Year 3 300
Customers in Year 4 1,200
Customers in Year 5 4,800
Customers in Year 6 14,400
Customers in Year 7 43,200
Customers in Year 8 129,600
Customers in Year 9 129,600
Customers in Year 10 129,600
EDD Score 34

27. Name of Product Score
Product A 533
Product B 483
Feature A 160
Feature B 150
Product D 118
Product E 37
Compare Scores

28. Most of the value is in those conversations

29. Can you trust the EDD Scores?

30. Turning Your Gut Feel into Data
EDD ROADMAP
Product
Description
Overview of the
product idea.
Similar to an exec
summary.
Score
Spreadsheet for
calculating ROI of
a product
Roadmap
Includes all
activities needed
to build and ship
the product.

31. With the EDD score sheet and
product description complete,
build your case.
Create experiments for each
metric
Scope each experiment
Do the ones with the best yield
first
Product Experiment Roadmap
Gaining confidence in the numbers you have placed in your EDD Score

32. Validating your assumptions
Hypothesis Experiment Results Conclusions
Target your metrics
Document &
discuss your
findings
Confirm/change your
assumptions. Publish
findings.
Create and run
experiments

33. Start with your gut, decide with data
Hypothesis Experiment Results
Users will value and
pay for a bookkeeper
on demand.
Ask 50 customers,
bookkeepers and
partners if they would
be willing to pay for
this service.
27 of 50 respondents
said they would be
willing to pay for online
support from a
bookkeeper for up to
60 minutes.
Conclusions
Now that we know a
realistic attach rate, we
can better calculate the
ROI on this effort.

34. Anatomy of an experiment
What is <our cost to build, revenue per user, etc.>
We will <describe experiment>
Then we will see if <Hypothesis>
We believe <Predict Result>
This means … and we can be more / less confident
in our critical question
Critical Question
Experiment
Observe
Hypothesize
Result

35. Experiment Definition
1. What we will do
to conduct the
experiment.
2. The data that we
will gather and how
we will analyze it.
3. What we expect
the data to show.
4. How we will interpret
the data if it is what we
expect.
"We will....." "Then we will see if…” "we believe...." "and this means...." + "...so we
can be more confident <critical
question>

36. 1. We will
research
competitors Ys
pricing offering
for feature X
2. Then we will see if
our planned pricing
is less expensive.
3. We believe our
planned pricing is less
than theirs...
4. and this means that
customers will pay $1 per
for transaction so we can
be more confident in our
monthly revenue per user
critical number.
Experiment Definition
4 key sections

37. Certainty
Hours
Lag
Total Cost
Cost of Certainty
Pricing/costing experiments
Units
1 - 10
Hours
Days
Hours * $100 + Days * $1,000
Total Cost / Certainty

38. Execute in order of Cost of Certainty

39. Idea Designed Working
Waiting
on Data
Doc.
Findings
Debate
Findings
Publish
Findings
Experiment Process

40. Tracking Experiments
LHlhjkTeam Experiment Board
`
`
`

41. Document, Debate,
Share
Results - What was the data you
generated?
Experiment Quality - Did you
gain any useful data?
Budget - Did it take you as long
as you expected?
Confidence Increase - Did your
confidence go up? How much?
Share the results on company
intranet if interesting or insightful
Did the model change?

43. Thanks
Joshuah Vincent
@joshuahvincent
joshuahv.com