Hypothesis Testing Workshop (cezary.co)
•
3 j'aime•1,666 vues
This is a presentation and workshop I shared during the launch of Intel Capital's EdTech Accelerator. It covers two core topics: 1) Developing and prioritizing strong hypothesis statements 2) Testing hypotheses via qualitative and quantitative tools My thinking draws draws on Steve Blank's customer development process and Javelin's Experiment Board and enhances it with the startup marketing process I've written about on my blog (http://www.cezary.co). The workshop is relevant to both technology companies as well as any startups looking to validate their business.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (20)
Similaire à Hypothesis Testing Workshop (cezary.co)
Similaire à Hypothesis Testing Workshop (cezary.co) (20)
Plus de Cezary Pietrzak
Plus de Cezary Pietrzak (8)
Dernier
Dernier (20)
Hypothesis Testing Workshop (cezary.co)
- 1. HYPOTHESIS TESTING WORKSHOP CEZARY PIETRZAK Intel EdTech Accelerator Redwood City, California August 4, 2015 @ckp
- 2. MY BACKGROUND 2000 - 2004 2005 - 2008 CEZARY & CO 2008 - 2012 2012 2013 2014
- 3. AGENDA WORKSHOP (1:30PM - 4:30PM) Developing hypotheses (25min) Exercise: Create your own (60min) Break (10min) Testing hypotheses (15min) Exercise: Deploy a test (70min)
- 4. HYPOTHESIS STATEMENTS ARE DECLARATIVE Definition States a fact or an argument Ends with a full stop Example I believe that…(X is true)
- 5. THEY BEGIN WITH THE CUSTOMER-PROBLEM Customer-Problem Customer X has Problem X (that is worth solving) I believe that… High school students struggle with basic data literacy *Always phrase the problem from the customer’s perspective *Add emotional language whenever possible
- 6. AND THEN ADDRESS THE PROBLEM-SOLUTION Problem-Solution Problem X can be solved by Solution X I believe that… Bringing real-life data into the classroom will make math and science more engaging *Only relevant after you’ve validated that a problem exists *Solution should be unique to alternatives
- 7. STRONG HYPOTHESES ARE DERIVED FROM DATA & INSIGHTS Customer-Problem High school students struggle with basic data literacy Rationale 25M high school students are falling behind in quantitative subjects Problem-Solution Bringing real-life data into the classroom will make math and science more engaging Rationale The explosion of data today makes it more accessible than ever before
- 8. AND ARE ALWAYS BASED ON ASSUMPTIONS Customer-Problem High school students struggle with basic data literacy Assumption Parents and school districts consider math and science a high priority Problem-Solution Bringing real-life data into the classroom will make math and science more engaging Assumption Most students are genuinely interested in math and science *Focus on the riskiest assumption
- 9. MANY EARLY-STAGE STARTUPS TEST CORE HYPOTHESES WITH THE EXPERIMENT BOARD
- 10. BEFORE MOVING TO MORE GRANULAR TESTING Customer Overarching persona Specific segment Stage in lifecycle Acquisition source Geography/location Users vs. influencers vs. decision-makers Problem / Challenge Master problem Sub-problem Distribution/marketing channels Buying process/sales funnel Willingness to pay/pricing Messaging Loyalty/retention Evangelism/virality Solution Existing product/service New product/service Specific feature/flow Marketing tactic *Keep one variable constant
- 11. YOU CAN SOURCE HYPOTHESES FROM MANY PLACES Behavioral insights Background research Industry benchmarks Company stage Growth goals Investor requirements Previous hypotheses
- 12. Highest risk Biggest potential impact Start high-level, then get specific Questions to ask Is the hypothesis core to your value proposition? Will validating the hypothesis significantly de-risk your business? For each hypothesis, have you validated the higher-level one above it? BUT MUST PRIORITIZE THEM TO CREATE VALUE
- 13. LAUNCH EXPERIMENTS IF THEY MEET 3 CRITERIA Speed “one-afternoon” rule simple over elaborate Cost cheap tools sweat equity shared resources Accuracy right audience right approach
- 14. SET GOALS AND SUCCESS METRICS Goals should be quantitative when possible Qualitative results can be measured Examples A minimum of X respondents will agree The majority of respondents will exhibit positive language User will convert at a 5% significance level Overall retention will rise by X% 1 in 5 people will recommend a friend
- 15. FOLLOW THESE BEST PRACTICES Baseline Better Keep it simple Customers understand the detailed technology behind my product Customers understand why my product is faster than competition Break big hypotheses into smaller batches I can acquire users via paid spend I can acquire users via social ads I can acquire users via paid content I can acquire users via trade shows Add details where possible I can acquire new users from Facebook efficiently I can acquire new users from Facebook mobile ads at $1/click Test one thing at a time (avoid: and/either/or) Customers prefer product X over product Y or Z Customers prefer product X over Y Customers prefer product X over Z
- 16. BREAKOUTS (60 minutes) Define core customer-problem & problem-solution Compile & prioritize list of key hypotheses Share work with group bitly.com/hypothtesting
- 17. PART 2 (15 minutes) Testing hypotheses
- 18. START WITH QUALITATIVE TESTING TO GATHER FEEDBACK In-person conversations / live demos Focus groups Video conversations User testing / video recording Public forums / blogs Surveys ($ or no $)
- 19. FOCUS THE CONVERSATION FOR BETTER RESULTS Don’ts Do’s Don’t rely on friends and family Do speak with outsiders who lack a filter Don’t tell respondents you’re working on an idea Do tell respondents that you’re seeking an opinion Don’t ask leading questions Isn’t it terrible when…? Do let them describe their environment and motivations Don’t provide options unless necessary Is your preference A, B or C? Do let people decide on the answer Don’t ask hypotheticals What would you do if…? Do ask about past behavior How have you solved this problem in the past? Don’t ask about intent Would you tells others about this product? Do test their willingness to take action Can you share this product with a friend?
- 20. USE QUANTITATIVE TESTING TO CAPTURE KEY DATA AND IDENTIFY ACTION Product analytics Heat maps Social media ads Paid search Landing pages Placeholders / waitlists A/B testing
- 21. ENSURE THAT YOUR DATA IS RELIABLE Questions to ask Who is your audience? How did they come to your app? Are the results real or sampled? Is the sample representative of your audience? How is significance determined?
- 22. EVALUATE RESULTS Significance of sample Minimum 10 responses for qual, 30 for quant Tangible outcome Signed letter of intent, verbal agreement Strong body language Overall enthusiasm, leaning in Correct metric hierarchy Later funnel metrics trumps early funnel If data is ambivalent, revisit hypothesis
- 23. BREAKOUTS (70 minutes) Define your experiment Choose tools Start implementing
- 24. LEARN ABOUT MARKETING + GROWTH + MOBILE AT CEZARY.CO @ckp