Lean Analytics workshop (from Lean Startup Conf)

Lean Analytics
Use data to build a
better business faster.

www.leananalyticsbook.com
@byosko | @acroll
@leananalytics

Who you are, who we are, what
we’ll cover

Case studies to explore
E- 2-sided Mobile User-gen
SaaS Media
commerce market app content

Empathy

Stickiness

Virality

Revenue

Scale

Plus...

All attendees Hashtag

12 clicks

28 responses 3 responses

Most startups don’t know what they’ll be
when they grow up.

Freshbooks Mitel
was invoicing Wikipedia was a
Paypal
for a web was to be lawnmower
ﬁrst built for company
design ﬁrm written by
Palmpilots experts only

Flickr
Hotmail Twitter Autodesk
was going to
was a was a made desktop
be an MMO
database podcasting automation
company company

Kevin Costner is a lousy entrepreneur.

Don’t sell what you can make.
Make what you can sell.

Analytics is the measurement
of movement towards your
business goals.

http://www.ﬂickr.com/photos/itsgreg/446061432/

Analytics, TL;DR

ATTENTION ACTIVATION RETENTION REVENUE

CONVERSION
NEW
RATE
VISITORS

GROWTH PAGES x
SEARCHES PER
VISIT RETURNS GOAL VALUE
TWEETS NUMBER
CONTENT
MENTIONS OF VISITS
TIME ETC.
ADS SEEN ON WORD OF
LOSS SITE MOUTH
BOUNCE
RATE REFERRAL

In a startup, the purpose of
analytics is to iterate to a
product/market ﬁt before
the money runs out.

What makes a good metric?

• It’s comparable to another time period, group, competitor, etc.
• It’s understandable in a way the target audience will understand
• It’s a ratio or rate
• Which means it’s easier to act on (acquisition cost per customer)
• It allows you to represent the tension between two things (ads shown versus
bounce rate, for example)
• It’s targeted to the right audience (Internal business, developers, marketers,
investors, media)
• It changes the way you behave
• “Accounting” metrics make your predictions more accurate
• “Experimental” metrics make your future behavior more effective

Think about a car

• You know 60MPH is twice as fast as 30MPH

• In a country, speed limits and mileage are well understood

• Kilometers are conveniently decimal; miles map to hours

• Miles travelled is good; miles per hour is better; accelerating or decelerating
changes your gas pedal

• You can measure “MPH divided by speeding tickets” as a metric of “driving fast
without losing my license”

Vanity metrics
A metric from the early, foolish days of the Web. If you have a site with many objects on
Hits it, this will be a big number. Count people instead.

Only slightly better than hits, since it counts the number of times someone requests a
Page views page. Unless you’re displaying ad inventory you should count people instead.

Is this one person visiting a hundred times, or are a hundred people visiting once? Fail.
Visits

The only thing this shows you is how many people saw your home page. It tells you
Unique visitors nothing about what they did, why they stuck around, or if they left.

Counting followers rather than actions is a bad idea. Once you know how many
Followers/friends/likes followers will do your bidding when asked, you’ve got something.

Time on site, or pages Poor substitute for actual engagement or activity. If customers spend a lot of time on
per visit your support or complaints pages, that could be a bad thing.

Until you know how many will open your mails (and act on what’s inside them) this isn’t
Emails collected useful. Test some of them and see.

While it sometimes affects your place in app stores and rankings, downloads alone
Number of downloads don’t lead to lifetime value. Measure activations, account creations, or something else.

5 essential dimensions in analytics

Qualitative Quantitative
Unstructured, anecdotal, revealing, hard to Numbers and stats; hard facts but less insight.
aggregate.

Vanity Actionable
Make you feel good, but don’t change how you’ll Change your behavior by helping you pick a
act. course of action.

Exploratory Reporting
Speculative, trying to ﬁnd unexpected or interesting Predictable, keeping you abreast of normal,
insights. managerial operations.

Leading Lagging
Number today that shows metric tomorrow— Historical metric that shows how you’re doing—
makes the news reports the news

Correlated Causal
Two variables that change in similar ways , perhaps An independent factor that directly impacts a
because they’re linked to something else dependent one

Donald Rumsfeld on analytics

Are facts which may be wrong and
we know should be checked against data.

know
we don’t Are questions we can answer by
reporting, which we should baseline
know & automate.
Things we
Are intuition which we should
we know quantify and teach to improve
don’t effectiveness, efﬁciency.
know
we don’t Are exploration which is where
unfair advantage and interesting
know epiphanies live.
(Or rather, Avinash Kaushik channeling Rumsfeld)

Segments, cohorts, A/B, and multivariates

Cohort:
Comparison of
similar groups
along a timeline.

Segment: A/B test: ☀ Multivariate
Cross-sectional ☀ Changing one analysis
comparison of all thing (i.e. color)
☁ Changing several
people divided by and measuring ☀ things at once to
some attribute
☁ the result (i.e. see which correlates
☁
(age, gender, etc.) revenue.) with a result.

Why use cohorts? Here’s an example.

Averages January February March April May
hide the
pattern Rev/customer CA$5.00 CA$4.50 CA$4.33 CA$4.25 CA$4.50

Cohort 1 2 3 4 5
Cohorts
show the January CA$5 CA$3 CA$2 CA$1 CA$0.5
revenue
drop as February CA$6 CA$4 CA$2 CA$1
customers
age March CA$7 CA$6 CA$5

April CA$8 CA$7

May CA$9

Averages CA$7 CA$5 CA$3 CA$1 CA$0.5

A few words on causality

http://www.ﬂickr.com/photos/roryﬁnneren/65729247

50

37.5

25

12.5

0
1 2 3 4 5 6 7 8 9 10
Seat rentals
http://www.rvca.com/anp/wp-content/plugins/wp-o-matic/cache/57226_07+proof+1a+hb+beach+day.jpg

http://www.imdb.com/media/rm3768753408/tt0073195

http://www.ﬂickr.com/photos/kapungo/2287237966

10000

1000

100

10

1
Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec

Ice cream consumption Drownings

http://www.ﬂickr.com/photos/25159787@N07/3766111564

http://www.ﬂickr.com/photos/wheressteve/3284532080

http://www.ﬂickr.com/photos/wtlphotos/1086968783

10000

1000

100

10

1
Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec
Ice cream consumption Drownings Temperature

http://www.ﬂickr.com/photos/stuttermonkey/57096884

http://www.ﬂickr.com/photos/germanuncut77/3785152581

http://www.ﬂickr.com/photos/fasteddie42/2421039207

Mostly, you’re easily distracted

Startup founder

One metric to rule them all
After all this, you should focus on one metric.

The right information
in the right place
just changes your life.

Stewart Brand

The right information
in the right place
at the right time
just changes your life.

Stewart Brand

Two dimensions show you
the One Metric That Matters
SaaS Media

Business model
Empathy

Stickiness
Stage

Virality

Revenue

Scale

An example you might not
have expected.

• Stage: Revenue
• Model: Retailer
• Solare is an Italian ﬁne-dining restaurant under new management. The new team
is trying to identify the key metrics and leading indicators

Solare watches the numbers

• One Metric That Matters: Gross Revenue to Labor Cost
• Under 30% is good
• Below 24% is great
• Lower than 20% and you may be under-stafﬁng, leading to dissatisﬁed
customers
• A leading indicator to optimize: Total covers is 5x reservations at 5PM
• If you have 50 reservations at 5, you’ll have 250 covers that night.
• This ratio varies by restaurant.

• If the reservations cause the covers, then focusing on more reservations will
grow the business. How would they test this?

The 5 stages of Lean Analytics

Dave McClure’s Pirate metrics

How do your users become aware of you?
Acquisition
AARRR
SEO, SEM, widgets, email, PR, campaigns, blogs ...

Do drive-by visitors subscribe, use, etc?
Activation Features, design, tone, compensation, affirmation ...

Does a one-time user become engaged?
Retention Notifications, alerts, reminders, emails, updates...

Do you make money from user activity?
Revenue Transactions, clicks, subscriptions, DLC, analytics...

Do users promote your product?
Referral Email, widgets, campaigns, likes, RTs, affiliates...

Eric Ries’
Three engines

Stickiness Virality Price

Approach Keep people Make people Spend revenue
coming back invite friends getting customers

Math that Get customers How many they Customers are
matters faster than you tell, how fast worth more than
lose them they tell them they cost to get

Long Funnel

• Inject signal at the top

• Measure results at the bottom

• Includes multi-channel interactions

Multi-channel funnels overview

Ash Maurya’s
Lean canvas

Lean Canvas box Some relevant metrics
Respondents who have this need; respondents who are aware of having
Problem the need.
Respondents who try the MVP; engagement; churn; most-used/least-
Solution used features; people willing to pay.
Feedback scores; independent ratings; sentiment analysis; customer-
Unique Value Proposition worded descriptions; surveys; search and competitive analysis.
How easy it is to find groups of prospects; unique keyword segments;
Customer Segments targeted funnel traffic from a particular source.
Leads/cust per channel; viral coefficient and cycle; net promoter score;
Channels affiliate margins; open, click-through rate; PageRank; message reach.
Respondents’ understanding of the USP; patents; brand equity; barriers
Unfair Advantage to entry; number of new entrants; exclusivity of relationships.
Lifetime customer value; Average revenue per user; Conversion rate;
Revenue Streams Shopping cart size; Click-through rate.
Fixed costs; cost of customer acquisition; cost of servicing the nth
Cost Structure customer; support costs; keyword costs.

Sean Ellis’
Startup growth pyramid

Step on the gas in
Scale new markets,
growth products, channels.
Find a defensible
Stack the odds unfair advantage
and tweak it.

Decide what you
Product/market ﬁt sell to whom, then
prove it.

McClure! Sean Ellis
Maurya Lean Lean Lean Analytics
Pirate Growth
Canvas! Startup!
Metrics! Pyramid! phase!

Problem Acquisition!
Problem
validation
(of testers,
prospects, Empathy
Customer Solution
etc.)
segments
validation
Product/
market ﬁt
Unique value MVP building
proposition
Activation
MVP Stickiness
Startup lifecycle stage!

Solution
iteration, Retention
sticky engine

(Natural)
channels
Organic
Stacking the
growth, viral Referral
Virality

Growth rate
engine
odds

Revenue
stream
Monetization,
price engine
Revenue
Revenue
Cost
structure
Scale
(Formal) growth
channels
Inorganic Attention!
growth, (at scale, of Scale
Unfair beyond Lean
customers)
advantage

Pause #1
What stage are you at?
What model are you in?
If you could have only one number what would it be?

Business model ﬂipbook
Revenue model: How you take money from someone
Product type: What you give them in return Together, these
Delivery model: How you get it to them make up a
Acquisition channel: How they learn about you business model
Selling tactic: How you convince them to buy

Paid advertising Banner on Informationweek.com
Search Engine Mgmt. High pagerank for ELC in kid’s toys
Acquisition Social media outreach Active on Twitter i.e. Kissmetrics

channel
How the visitor, Inherent virality Inviting team member to Asana
customer, or user finds
out about the startup. Artificial virality Rewarding Dropbox user for others’ signups
Affiliate marketing Sharing a % of sales with a referring blogger
Public relations Speaker submission to SXSW
App/ecosystem mkt. Placement in the Android market

Simple purchase Buying a PC on Dell.com
What the startup does Discounts & incentives Black Friday discount, loss leader, free ship
Selling
tactic

to convince the visitor Free trial Time-limited trial such as fitbit Premium
or user to become a Freemium Free tier, relying on upgrades, like Evernote
paying customer.
Pay for privacy Free account content is public, like Slideshare
Free-to-play Monetize in-app purchases, like Airmech
One-time transaction Single purchase from Fab
How the startup Recurring subscription Monthly charge from Freshbooks
Revenue
model

extracts money from its Consumption charges Compute cycles from Rackspace
visitors, users, or
Advertising clicks PPC revenue on CNET.com
customers.
Re-sale of user data Twitter’s firehose license
Donation Wikipedia’s annual campaign

Software Oracle’s accounting suite
What the startup does Platform Amazon’s EC2 cloud
in return. May be a
Product

Merchandising Thinkgeek’s retail store
type

product or service; may
be hardware or User-generated content Facebook’s status update
software; may be a Marketplace AirBnB’s list of house rentals
mixture. Media/content CNN’s news page
Service A hairstylist
Delivery

Hosted service Salesforce.com’s CRM
model

How the product gets Digital delivery Valve purchase of desktop game
to the customer. Physical delivery Knife shipped from Sur La Table

Business Flipbook
Dropbox example
aspect page(s)

Acquisition Inherent virality. Sharing ﬁles with others.
channel Artiﬁcial virality. Free storage when others sign up.

Selling Limited-capacity accounts are free;
Freemium.
tactic subscribe when you need more.

Revenue Recurring $99/year, monthly fees, enterprise
model subscription. tiers.

Product Storage-as-a-service with APIs,
Platform.
type collaboration, synchronization tools.

Delivery Hosted service. Cloud storage, web interface.
model Digital delivery. Desktop client software.

E-commerce
TL;DR:
• Are you focused on loyalty or acquisition?
• Pricing matters more than you think
• Don’t overlook logistics, delays, and ratings
• Old “average conversion rates”

Loyalty or acquisition?

How many of
your customers Then you are in Your customers You are just
Focus on
buy a second this mode will buy from you like
time in 90 days?

Low CAC,
1-15% Acquisition Once 70% high
of retailers checkout

15-30% Hybrid 2-2.5 20% Increasing
per year of retailers returns

Loyalty,
>30% Loyalty >2.5 10% inventory
per year of retailers expansion
(Thanks to Kevin Hilstrom for this.)

Pricing has a huge impact

http://hbr.org/1992/09/managing-price-gaining-proﬁt/ar/8

Don’t forget the real world

Shipping time, stock availability,
logistics, ratings, and other factors
have a real impact on most e-
commerce companies.

E-commerce model:
WineExpress increases
revenues

• Stage: Revenue
• Model: E-commerce
• Exclusive wine shop partner of the Wine Enthusiast catalog and website.
• “Wine of the day” page is highly trafﬁcked, needed optimization

WineExpress: before and after

• Tested 3 design variations, primarily focused on layout

• One version was a clear winner

• Conversion went up for a heavily discounted shipping promotion

• Real key was a 41% increase in revenue per visitor

Software-as-a-Service
TL;DR:
• Eventually, focus on Customer Acquisition Payback
• Engagement varies by intended use of the app (i.e. CRM versus travel booking)
• Credit cards up front have a huge effect
• Freemium is a sales tactic, not a business model
• Churn, acquisition cost, and lifetime value
• Subscriptions may be a bad thing (per-transaction pricing is an option too.)

Some sample churn calculations

• Active users have logged in at least once in the month after signing up
• New users are growing at 20% a month
• 30% use the service at least once (in the month after signing up)
• 2% convert into paid customers

• Example churn calculation for February
• Lost / Starting with (Paying Users) * 100
• 26 / 1035 * 100 = 2.5%
• If 2.5% of customers churn every month, it means that the average customer
stays around for 40 months (100 / 2.5).
• This is how you can start to calculate the Lifetime Value of a customer (40
months * Cost of the Service.)

SaaS model:
Backupify’s customer
lifecycle

• Stage: Scale
• Model: SaaS
• Leading backup provider for cloud based data.
• The company was founded in 2008 by Robert May and Vik Chadha
• Has gone on to raise $19.5M in several rounds of ﬁnancing.

Shifting to Customer Acquisition
Payback as a key metric

• Initially focused on site visitors

• Then focused on trials

• Then switched to signups

• Today, MRR

• In early 2010, CAC was $243 and ARPU was only $39

• Pivoted to target business users

• CLV-to-CAC today is 5-6x

• Now they track Customer Acquisition Payback

• Target is less than 12 months

Mobile
TL;DR:
• App stores make or break you
• In-app revenue is the only way to make a living

Not all churn is equal

“Track churn at 1 day, 1 week and 1 month, because users
leave at different times for different reasons. After one day it
could be you have a lousy tutorial or just aren’t hooking
users. After a week it could be that your game isn’t ‘deep
enough,’ and after a month it could be poor update
planning.”
Keith Katz, co-founder of Execution Labs
and former VP of Monetization for OpenFeint

(Knowing when users churn gives you an indication of why they’re churning and
what you can try in order to keep them longer.)

Mobile app model:
Localmind hacks Twitter

• Stage: Empathy
• Model: UGC/mobile
• Real-time question and answer platform tied to locations.
• Needed to ﬁnd out if a core behavior—answering questions about a place—
happened enough to make the business real

Localmind hacks Twitter

• Before writing a line of code, Localmind was concerned that people would never
answer questions.
• This was their biggest risk: if questions went unanswered users would have a
terrible experience and stop using Localmind.
• Ran an experiment on Twitter
• Tracked geolocated tweets in Times Square
• Sent @ messages to people who had just tweeted, asking questions about
the area: how busy is it; is the subway running on time; is something open;
etc.
• The response rate to their tweeted questions was very high.
• Good enough proxy to de-risk the solution, and convince the team and
investors that it was worth building Localmind.

Media
TL;DR:
• Advertising is a complicated, many-faced beast

Paywalls and baselines

• Paywalls might work
if done right. Jury’s
still out.

• A June, 2012 study by the Advertising Research Foundation conducted across a
half-million ad impressions showed that blank ads bearing no information had a
click-through rate of roughly 0.08%—comparable to that of some paid
campaigns.

Media model:
Just For Laughs and
Youtube

• Stage: Revenue
• Model: Media
• Gags program launched in 2000 shows visual pranks with no words, licensed
widely through traditional TV channels. Recently, the organization learned about
its popularity on YouTube and decided to invest time in a channel.

Just for Laughs Gags

• Had been a YouTube partner since 2009, but intended to build its own site
• Noticed some unauthorized activity online, so chose to bulk upload 2,000 2-
minute prank clips.
• Experimented with overlay, pre- and post-roll ads. Short format meant intro video
clips drove people away.
• 10-15 second intro resulted in a 30% drop-off.
• Tested longer versus short-form videos
• In 24h, both long- and short-form clips got 30-40K views.
• Long-form clip had 5x that of a short clip—but a long clip had 12 short ones,
so overall the long one pays less.
• Long-form video has a longer viewing tail.
• Long-form has intro, so there’s a 40% audience drop-off halfway into an
episode, versus a 15% drop-off halfway into a single short video.

Just for Laughs Gags

• Big growth in UGC since the focus on YouTube
• Could have done a DCMA takedown but decided to monetize instead
• Less lucrative than on-channel content, but volume makes up for it
• 100,000 user-generated videos generate 40-50% of total monthly views
• 2h mash-ups can get millions of views
• Today, JFL tries to learn from and emulate what users show them is working

User-generated content
TL;DR:
• Content virality and user virality
• Trying to move users up the engagement funnel
• More time on fraud than you expect
• Notiﬁcations and email are the real user interface
• Passive content creation is on the horizon

Notifications are the real UI

“… notifications become the primary way I use the
phone and the apps. I rarely open twitter directly. I see that
I have ‘10 new @mentions’ and I click on the notification
and go to twitter @mention tab. I see that I have ‘20 new
checkins’ and I click on the notification and go to the
foursquare friends tab…”
- Fred Wilson

• Allows use of many more engagement apps on my phone without them being on
the main page
• Have as many communications apps as I want. Don’t need to use the apps, just
the notification inbox
• Notification inbox is the new home screen

UGC model:
Reddit goes from links to
community

• Stage: Virality
• Model: UGC
• A graduate of the ﬁrst YCombinator class, reddit was acquired by Conde Nast
but left largely to its own devices. Thanks to a vibrant community and some
good guidance by its founders, it’s a trafﬁc powerhouse.

Reddit goes from links to community

• Product evolution
• Started as a simple link-sharing site with voting
• Then added the ability to comment, with votes on comments
• Then created the ability to make “self-posts” rather than only comment on off-
site trafﬁc
• Now self-posts are more than half of all posts
• Revenue from ads and “reddit gold”
• Started as a joke, but turned into a revenue source
• One person paid $1000 for a month; some paid $0.01. Avg. around $4.
• Paying users get early access to features, since they’re an engaged beta
• Spam is a big issue
• Human ﬂagging isn’t good enough
• 50% of the company’s development time is devoted to beating spammers
• Having a 7y “training set” helps them develop and test better algorithms

2-sided market
TL;DR:
• Focus on the money side
• Breaking the chicken-and-egg

Plenty of examples

• Real estate listing services
• Crowdfunders like Indiegogo and Kickstarter
• Charities like Donors Choose
• Seller markets like eBay, Craigslist, and Etsy
• App stores
• Dating sites
• Excess inventory travel like Hotwire and Priceline

• They all:
• Include a shared inventory model
• Have two stakeholders—buyers and sellers; creators and supporters;
prospective partners; or hotels and travellers
• Make money when the two stakeholders come together
• Differentiate based on a particular set of search parameters or
qualiﬁcations (apartments that have been vetted; seller ratings.)
• Need an inventory to get started

Chickens and eggs

Seed the buyer side: demand Seed the seller side: artiﬁcial
creation inventory
• When Uber launched in Seattle, it • Amazon started selling books; then
paid drivers $30 an hour to drive broadened its offering; then launched
passengers around a marketplace for goods from many
• Only switched to a commission other suppliers.
model once they had enough • They created inventory; then
demand to make it worthwhile for the demand; and then a two-sided
drivers. marketplace.

• Knew when to switch by: • Knew when to switch by:
Measuring how much drivers would Measuring loyal buyers; identifying
be making on a commission basis, unmet search needs and pent-up
as well as the inventory and the time demand.
it took a driver to pick up a customer.

2-sided market model:
AirBnB and photography

• Stage: Revenue
• Model: 2-sided marketplace
• Rental-by-owner marketplace that allows property owners to list and market
their houses. Offers a variety of related services as well.

AirBnB tests a hypothesis

• The hypothesis: “Hosts with professional photography will get more business.
And hosts will sign up for professional photography as a service.”

• Built a concierge MVP

• Found that professionally photographed listings got 2-3x more bookings than the
market average.

• In mid-to-late 2011, AirBnB had 20 photographers in the ﬁeld taking pictures for
hosts.

NIGHTS BOOKED

10 million

8 million

6 million

20 photographers
4 million

2 million

2008 2009 2010 2011 2012

Pause #2
Which models would you like to see more about?

Lifecycle stage

Discover a known need within a sizeable market

Identify a solution reachable people will pay for

Develop and validate a viable product to sell proﬁtably

Create a sustainable business model

Attract and appease investors

Find a successful exit

Where is the risk?

Real need? Key: Empathy

Right solution?
Key: Stickiness

Key: Growth rate
Good product?
Key: Virality
Sustainable biz?
Key: Revenue
Healthy market?

Successful exit? Key: Scale

Lean Analytics “Gate” needed! Key metrics for
this stage! Rationale!
stage! to move forward!
I’ve found a real, poorly-met Qualitative responses
need a reachable market Identifying a real problem and a real solution is
Meetings held
Empathy
faces.
The cheapest thing to do (since it’s just the
Scored qualitative
price of coffee.) It also addresses the riskiest
I’ve figured out how to solve Surveys, bulk feedback
question—will anyone care? It comes first.
the problem in a way they
will adopt and pay for.
Signup rates
Will the dogs eat the dog food? Make your
Activation on invite/beta
Stickiness
mistakes with a small, friendly audience you
MVP adoption
can love and nurture before throwing the
I’ve built the right product/
features/functionality that Retention, churn
unwashed masses at it.
keeps users around.
Message open rates
Inherent sharing
Sharing helps grow, but also verifies that what
Virality
you’ve made is good. Word of mouth is
Growth rate

Viral coefficient/cycle time
endorsement. And virality is a “force
The users and features fuel
growth organically and Word of Mouth sharing
multiplier” for paid customer acquisition.
artificially.
Key goal conversion rates
Will people open their pocketbooks? And can
Customer lifetime value
you charge them enough to fund your
Revenue
I’ve found a sustainable, Customer acquisition cost
ongoing operations, plus your artificial
scalable business with the Margins
acquisition of users?
right margins in a healthy
ecosystem.
Access to employees
Access to capital
You need channels to amortize the cost of
Scale
Competition
sales and distribution. You need an
I can achieve a successful ecosystem to cross the “hole in the middle”
exit for the right terms.
Visibility/SEO/SEM
from niche player to big company
Channel health

Exit!

Empathy
You need to get inside your target market’s head. You need to know you’re solving
a problem people care about. This is really risky, so getting out of the building
makes a lot of sense. Few people would dispute this.

Signs you’ve found a problem worth tackling

Good signs Bad signs

• They want to pay you right away • They’re distracted

• They’re actively trying to (or have • Their shoulders are slumped or
tried to) solve the problem in they’re slouching in their chair (poor
question body language)

• They talk a lot and ask a lot of • They talk a lot but it’s not about the
questions demonstrating a passion problem or the issues at-hand
for the problem (they’re rambling)

• They lean forward and are animated
(positive body language)

How to avoid leading the witness
Avoid biased wording, preconceptions,
Don’t tip your hand or a giveaway appearance. Word your
surveys carefully to be neutral.

Get them to purchase. Ask them to pay. Demand real introductions. Or ask them “how
Make the question real many of your friends would say X” to avoid self-effacement

Ask “why” several times. Leave lingering, uncomfortable pauses in the conversation
Keep digging and let them ﬁll them.

Have a colleague make notes of when they react, or of their body language.
Look for other clues

Creating an answers-at-scale campaign

• Know why you’re doing a survey in the ﬁrst place

Ask what existing Ask how customers Ask what kind of Test which tagline
brands come to try to ﬁnd a product money people or unique value
mind in an industry or service spend on a proposition
problem resonates best with
customers

Market alongside Help you plan Choose the
them? Address marketing Shape your pricing winning one, or
competitors? campaigns and strategy just take that as
Choose partners? choice of media advice



• Design the survey Demographic Quantiﬁable Qualitative,
segmentation answers to your open-ended
questions research problem feedback



• Design the survey

• Test it (you’ll always have mistakes)





• Send it out

Via your NW To a paid list As an ad campaign

Beware of Give the solution or
Beware of Audience plea Name the problem
respondent bias, unique value
spamminess, low
misrepresentation
open rates
of the larger market “Are you a single “Can’t sleep? We’re “Our accounting
mom? Take this brief trying to ﬁx that, and software automatically
survey and help us want your input.”) ﬁnds tax breaks. Help
address a big us plan the product
challenge.” roadmap.”





• Send it out Were you able to capture the attention of the
market? Did they click on your ads and links?
Which ones worked best?
• Collect the results
Are you on the right track? What decisions can
you now make with the data you’ve collected? By
• Analyze the data segment!
Will people try out your solution/product? How
many of your respondents were willing to be
contacted? How many agreed to join a forum
or a beta? How many asked for access in their
open-ended responses?

Empathy stage:
LikeBright’s mechanical
turk

• Stage: Empathy
• Model: 2-sided marketplace
• Early stage startup in the dating space that joined TechStars Seattle in 2011
• Initially rejected, saying, “We don’t think you understand your customer well
enough.”

Talking to a hundred single women in a
day

• Used Mechanical Turk and Google Voice to speak with 100 single women; paid
$2. The interviews lasted typically around 10-15 minutes.
• Simple interview script with open-ended questions, since he was digging into the
problem validation stage of his startup.
• Founder Nick Soman: “I was amazed at the feedback I got. We were able to
speak with one hundred single women that met our criteria in four hours on
one evening.”
• Went back to TechStars and got accepted.
• LikeBright’s website is now live with a 50% female user base, and recently raised
a round of funding.
• “Since that ﬁrst foray into interviewing customers, I’ve probably spoken with
over a thousand people through Mechanical Turk,”

How to score problem interviews
Teach the controversy

How to score problem interviews

• Did the interviewee successfully rank the problems you presented?

• Are they actively trying to solve the problems (or have they done so in the past)?

• Were they engaged and focused throughout the interview?

• Did they agree to a follow-up meeting or interview (to present your solution)?

• Did they offer to refer others to you for interviews?

• Did they offer to pay you immediately for the solution?

Stickiness
This comes from a good product. This is the same as the Eric Ries’ Stickiness
engine of growth. You need to ﬁnd out if you can build a solution to the problem
you’ve discovered. There’s no point promoting something awful if your visitors will
bounce right off it in disgust. Companies like Color that attempted to scale
prematurely, without having proven stickiness, haven’t fared well.

1995 Hits
1997 Visits
1999 Visitors
2002 Conversions Who did you add? Where from? Why?
2010 Engagement What did they do? How did it beneﬁt?

Who did you lose? Why did they leave?

What time on page
can tell you about goals & behaviors

• People spend ~1m on a page
when they’re engaged with it
• If a page has high trafﬁc and
low engagement, why are
people leaving:
• Did they come expecting
something else?
• Is the layout working?
• Is it simply a page that
isn’t designed to keep
users for long?
• Show off your good stuff.
If a page has a high
engaged time but few
visitors, promote it.

(Thanks to Chartbeat for the analysis.)

More interesting when broken
down by business model

(Thanks to Chartbeat for the analysis.)

Days since last visit

1200 25000
Number of users

January February
1000
20000
800
15000
600
10000
400
5000
200

0 0
1 2 3 4 5 6 7 8 9 Disengaged
Days since last engagement
(>10 days)

Watch out for the dumb stuff
There’s a large number of ways to break an MVP

How to stay focused on the right stuff

Virality
Once you’ve got a product or service that’s sticky, it’s time to use inherent virality—
word-of-mouth that’s tied into the use of the product. That way, you’ll test out your
acquisition and onboarding processes on visitors who are motivated to try you,
because you have an implied endorsement from an existing user. You can also
think of virality as a force multiplier for paid promotion—so you want to maximize
that multiplier before you start spending money on customer acquisition through
inorganic methods like advertising.

3 kinds of virality
•
Inherent virality is built into the product, and happens as a function of use.
•
Artiﬁcial virality is forced, and often built into a reward system.
•
Word of mouth virality is simply conversations generated by satisﬁed users.

Viral coefﬁcient

------------------------------------------------------
Get your free private email at http://www.hotmail.com
------------------------------------------------------

Viral coefﬁcient

v ≠ 1, pt = δp0 (1 – vt+1) / (1 – v) + p0

http://robert.zubek.net/blog/2008/01/30/viral-coefﬁcient-calculation/

Or simpler

x Viral - Churn & >
Users
coefﬁcient abandonment 1

How to calculate it

• First calculate the invitation rate, which is the number of invites sent divided by
the number of users you have.
• Then calculate the acceptance rate, which is the number of signups or
enrollments divided by the number of invites.
• Then multiply the two together.
• Consider, for example
• Your 2,000 customers have sent out 5,000 invitations during their lifetime on
your site.
• Your invitation rate is 2.5.
• For every ten invitations received, one gets clicked.
• Your acceptance rate is 0.1.
• Multiply the two, and you have your viral coefﬁcient: 0.25. Every customer you
add will add an addition 25% of a customer.

Don’t forget cycle time
How long until they invite someone?
• After 20 days with a cycle time of two days, you will have 20,470 users.
• If you half that cycle time to one day, you would have over 20 million users!

Virality stage:
Circle of Moms ﬁnds an
engaged market

• Stage: Stickiness
• Model: UGC
• Launched as Circle of Friends in 2007, it was a way for small groups to interact
atop Facebook’s platform; but when engagement wasn’t good enough, the
founders decided to dig deeper.

The problem: Not enough engagement

• Too few people were actually using the product

• Less than 20% of any circles had any activity after their initial creation

• A few million monthly uniques from 10M registered users, but no sustained
traction

What Circle of Moms found

• They found moms were far more engaged
• Their messages to one another were on average 50% longer
• They were 115% more likely to attach a picture to a post they wrote
• They were 110% more likely to engage in a threaded (i.e. deep) conversation
• Circle owners’ friends were 50% more likely to engage with the circle
• They were 75% more likely to click on Facebook notiﬁcations
• They were 180% more likely to click on Facebook news feed items
• They were 60% more likely to accept invitations to the app
• Pivoted to the new market, including a name change
• By late 2009, 4.5M users and strong engagement
• Sold to Sugar, inc. in early 2012

Revenue
You’ll want to monetize things at this point. That doesn’t mean you haven’t already
been charging—for many businesses, even the ﬁrst customer has to pay. It just
means that earlier on, you’re less focused on revenue than on growth. You’re
giving away free trials, free drinks, or free copies. Now you’re focused on
maximizing and optimizing revenue. This phase is closely tied to the Lean Startup’s
Price engine of growth.

Sergio Zyman’s many “mores”

• If you’re dependent on physical, per-transaction costs (like direct sales, or
shipping products to a buyer, or signing up merchants) then more efficiently
will figure prominently on either the supply or demand side of your business
model.
• If you’ve found a high viral coefficient, then more people makes sense, because
you’ve got a strong force multiplier added to every dollar you pour into customer
acquisition.
• If you’ve got a loyal, returning set of customers who buy from you every time,
then more often makes sense, and you’re going to emphasize getting them to
come back more frequently.
• If you’ve got a one-time, big-ticket transaction, then more money will help a lot,
because you’ve only got one chance to extract revenue from the customer and
need to leave as little money as possible on the table.
• If you’re a subscription model, and you’re fighting churn, then upselling
customers to higher-capacity packages with broader features to additional
subscribers within their organization is your best way of growing existing

What’s a customer worth?
BAD FOR YOU GOOD FOR YOU

Visitor User Customer
Spam/fraud
Support/resources
Ad impressions

Usage data

Content virality

Invite virality

Inherent virality

Voting/ﬂagging

Victim/critical mass

Revenue

Market/product ﬁt
Most people’s ﬁrst instinct when things aren’t going incredibly well is to build more
features. Instead, try pivoting into a new market.
• Assume the product isn’t the problem, it’s the target customer.
• It may be easier to change markets than products.

Scale
With revenues coming in, it’s time to move from growing your business to growing
your market. You need to acquire more customers from new markets. You can
invest in channels and distribution to help grow your user base, since direct
interaction with individual customers is less critical—you’re close to product/
market ﬁt and you’re analyzing things quantitatively. This phase is closely tied to
the acquisition cost optimization side of the Lean Startup’s Price engine of growth.

The hole in the middle

Differentiation Efﬁciency

Apple Costco
Here be
dragons

Your local
sustainable gluten-free
cupcake shop
Niche

Growth hacking
Finding your way up and to the right.
Human approaches, mathematical approaches

The leading indicator

• A Facebook user reaching 7 friends within 10 days of signing up (Chamath
Palihapitiya)
• If someone comes back to Zynga a day after signing up for a game, they’ll
probably become an engaged, paying user (Nabeel Hyatt)
• A Dropbox user who puts at least one ﬁle in one folder on one device (ChenLi
Wang)
• Twitter user following a certain number of people, and a certain percentage of
those people following the user back (Josh Elman)
• A LinkedIn user getting to X connections in Y days (Elliot Schmukler)

• These kinds of criteria also make great segments to analyze.

(from the 2012 Growth Hacking conference)

But wait: correlated or causal?

Correlation lets you Causality lets you
predict the future change the future

“I will have 420 “If I can make more
engaged users and ﬁrst-time visitors stay
75 paying customers on for 17 minutes I
next month.” will increase sales in
90 days.”

Optimize the
Find correlation Test causality
causal factor

The growth hack

• Growth hacking is simply what marketing should have been doing, but it fell in
love with Don Draper and opinions along the way

• At its most basic: Optimize a factor you think is correlated with growth

An example from Reddit

Logged-in users All users
Days since
last visit
Visits Pageviews Pg/visit Visits Pageviews Pg/visit

0 127,797,781 1.925B 15.06 242,650,914 3.478B 14.33

1 5,816,594 87,339,766 15.02 13,021,131 187,992,129 14.44

2 1,997,585 27,970,618 14 4,958,931 69,268,831 13.97
Causality: When we make
Correlation: If you look at 13.88
3 955,029 13,257,404
changes that get someone
2,620,037 34,047,741 13
>15 pages today you’ll
to look at 20,644,331
15 pages 12.32
they’ll
come back tomorrow. 14.23
4 625,976 8,905,483 1,675,476
come back.
5 355,643 4,256,639 11.97 1,206,731 14,162,572 11.74

Pause #3
What stage are you at?
What is your gating metric?

Asymptotes
What to do when there’s no line in the sand

How to draw the line yourself

• Sample company trying to drive enrollment
• At ﬁrst, out of over 1,200 visitors, only 4 (0.3%) sign up.
• By the end of the month, the site is converting 8.2% of its 1,462 visitors.
• Should the company keep optimizing?

How to draw the line yourself

• By plotting a trendline we see that in the current situation the line is around 9%.
• To really move the line will require a revolutionary, not evolutionary, change.

Finding your
One Metric That Matters

Your OMTM must ﬁt

Your basic The stage your
business model startup is at

(monetization) (lifecycle)
• E-Commerce • Empathy
• UGC • Stickiness
• Media • Virality
• SaaS • Revenue
• Mobile App • Scale
• 2-sided market

What’s your OMTM?
SaaS Media

Empathy Interviews; qualitative results; quantitative scoring; surveys

Loyalty, Inventory, Engagement, Downloads, Content, Traffic, visits,
Stickiness conversion listings churn churn, virality spam returns

CAC, shares, Inherent WoM, app Invites, Content
Virality reactivation
SEM, sharing
virality, CAC ratings, CAC sharing virality, SEM

(Money from transactions) (Money from active users) (Money from ad clicks)

Transaction, Transactions, Upselling, CLV, Ads, CPE, affiliate
Revenue CLV commission CAC, CLV ARPDAU donations %, eyeballs

Affiliates, Other API, magic Spinoffs, Analytics, Syndication,
Scale white-label verticals #, mktplace publishers user data licenses

In a startup,
focus is hard to achieve.

Having only one metric
addresses this problem.

Metrics are like
squeeze toys.

http://www.ﬂickr.com/photos/connortarter/4791605202/

B2B: Selling to the enterprise

Lean Analytics lifecycle
for an enterprise-focused startup

Stage Do this Fear this
Consulting to test ideas and Lock-in, IP
Empathy bootstrap the business control, overﬁtting

Standardization and integration; Ability to
Stickiness shift from custom to generic integrate; support

Word of mouth, references, case Bad vibes;
Virality studies exclusivity

Growing direct sales, professional Pipeline, revenue
Revenue services, support recognition, comp

Channels, analysts, ecosystems, Crossing the
Scale APIs, vertically targeted products chasm; Gorillas

Intrapreneur example:
P&G changes the mop
instead of the soap

• Stage: Empathy
• Model: Retail/consumer packaged goods
• P&G is constantly looking for better soaps. But innovation was slowing.
Frustrated, they hired a design team to help them.

P&G changes the mop
instead of the soap

• Heavy internal investment in R&D, but limited results
• Brought in an outside agency (Continuum) to help
• The team watched people as they mopped, recording and iterating their
research approach

• Watched someone pick up spilled coffee. Rather than mopping, the person
swept up with a broom, then wiped with a cloth
• Realized the mop, not the liquid, mattered
• Studied the makeup of ﬂoor dirt; realized much of it is dust

• Swiffer is a $500M innovation in a stalled industry

The Lean Analytics lifecycle
for an Intrapreneur
Stage Do this Fear this
Get buy-in Political fallout
Beforehand
Find problems; don’t test demand. Entitled, aggrieved
Empathy Skip the business case, do analytics customers

Know your real minimum based on Hidden “must haves”,
Stickiness expectations, regulations feature creep

Build inherent virality in from the Luddites who don’t
Virality start; attention is the new currency understand sharing

Consider the ecosystem, channels, Channel conﬂict,
Revenue and established agreements resistance, contracts

Hand the baton to others gracefully Hating what happens
Scale to your baby

In the end:
Ask good questions
The OMTM should answer the most important question.

Once, a leader convinced others in
the absence of data.

Now, a leader knows what
questions to ask.

Ben Yoskovitz
byosko@gmail.com
@byosko

Alistair Croll
acroll@gmail.com
@acroll

Lean Analytics workshop (from Lean Startup Conf)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Lean Analytics workshop (from Lean Startup Conf)

Similar to Lean Analytics workshop (from Lean Startup Conf) (20)

More from Lean Analytics

More from Lean Analytics (14)

Recently uploaded

Recently uploaded (20)

Lean Analytics workshop (from Lean Startup Conf)