Building a data-driven culture

Building a Data-Driven Culture
Olof Hoverfält
Director, Sanoma Games
September 2, 2015

Sanoma Games is a diverse yet compact and lean unit
B2C & B2B, web & mobile Autonomous unit Key functions “in-house”
Business development
Marketing
Design
Development
Sales

DESIGN
BUSINESS ANALYTICS
UX 
Business

Being data-driven requires much more than the right tools
http://toolmonger.com/category/85/computer/page/2/
Handheld
USB Snake
Scope

...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts

http://www.startupolic.com/launching-your-service-think-about-ux/; https://en.wikipedia.org/wiki/Lean_startup
An MVP is “a version
of a new product
which allows a team
to collect the
maximum amount of
validated learning
about customers with
the least effort"
The MVP approach is useful at all levels of development, from
single product feature to new business concepts
Adopt MVP thinking in all development

http://www.yesware.com/blog/how-to-close-more-deals-with-lean-email/
A continuous Build-Measure-Learn cycle enables fast learning
but is really sensitive to friction in workflow and cooperation
Ensure all key functions are fully dedicated and the workflow is frictionless

A data-driven culture cannot be created by coercion, but must
be built through intrinsic motivation from its visible benefits
Transparency Autonomy Ownership

xxx
xxx
xxx
xxx
xxx
xxx
xxx
xxx
xxx
xxx
Improvement actions
xxx
Revenue
ROS&packag.
campaign vol.
Video
campaign vol.
Pricing
Discounts
Campaign
volume
Optimization
Direct
Referral
SEO
Ret. visitors
Potential
issues
Site problems
Ad blockers
New visitors
Reach
Monetization
Visits
Premium
fill rate
Premium
eRPM
Remnant
eCPM
Unique
visitors
Media sales breakdown by contributing component
Impression
success rate
nn%
nn%
nn% 
mm%
nn%
nn%
nn%
lost
Ensure overall understanding of how the business works and of the contribution of and the
dependencies between its components. Make deep data available
In order for a team to develop and fine tune an engine,
everyone must know how it works

Adoption of a data-driven approach requires for people to see
its benefits; how they could kick ass even more at what they do
Image: http://blog.crazyegg.com/lessons/show-benefits/
Make the personal benefits of being data-driven visible.
Show how its autonomous pursuit enhances mastery.
Super You!
Data-drivenness
You

Ownership
is not about doing what I tell you to do
is not about not doing what I tell you not to do
is about doing what I don’t tell you to do
Ensure ownership of benefits, not deliverables
Focus on why, not what

Data-drivenness, at both individual and team level, is a
capability that can and should be developed
http://skepdic.com/tencommands.html
+
Encourage autonomous reflection but ensure conceptual guidance
Reflection Key concepts
Being data-driven is not only about
sensibly answering questions, but
about finding them in the first place
It cannot itself be standardized
into rules or processes, but it
can be guided by high-level key
concepts

A core principle of learning is to be hypothesis-driven. Random
experimentation will yield various outcomes but little insight
Make continuous hypothesis-driven split testing the standard approach in all development activities
across functions
Hypothesis:
[A statement that can
be shown true or false]
Design a test and a
metric that confirms
the hypothesis at a
certain pre set limit
Test in a real setting
with real usersPay attention also to
what is NOT
happening
Confirm or reject the
hypothesis based on
the chosen metric and
pre set limit
Add the learnings to
your body of
knowledge, also the
invalidated hypotheses
Build on the
knowledge to form the
next hypothesis

Data without insight can be destructive as it may lead
development the wrong way look like improvement
DAU
(daily active users)
Downloads
(i.e. not even activations)
Activity delta
(change in user engagement)
Churn
(users who quit)
>
Vanity metric
Indicator of
actual use
>Leading indicator of churn
Actionable
Lagging indicator
Not actionable
Put a lot of effort into designing KPIs (Key Performance Indicators) that are both sensible and
actionable; such that steer development initiatives toward your ultimate business goal

Main KPIs should be static but reflect the current growth phase,
while tactical tools can be whatever helps improve the KPIs
High-level (growth-stage-specific) KPIs to
continuously optimize
Tactical tools do explore user behavior and to
measure impact from change on KPI(s)
http://lanesplitter.jalopnik.com/x-ray-artist-reveals-the-inner-beauty-of-classic-motorc-1698564514

A simple tool that is actually used provides infinitely more value
than a sophisticated one that is not
https://xplr.com/xplr-umbrella-dataviz-on-top-of-unsupervised-machine-learning/
>
Make sure everyone has easy access to all tools they might need and know how to use them

Goals set at function level can often be unaligned or conflicting,
breaking joint effort and leading to sub-optimal performance
UX
Set high level common goals to ensure everyone works and cooperates to optimize the whole as
opposed to sub-optimizing their own components
Business Manager goal:
Visit eRPM
(effective revenue per session)
Product Manager goal:
Avg. DAU
(daily active users)

Example:
Continuous improvement and A/B testing

A/B testing is about continuous and systematic gradual
improvements that over time build increased performance
2.5x+
Improvement

Seemingly small but sustained improvements can have a
very large impact on overall performance
49%
Improvement
min to max
25%
Improvement
mean to max

Example:
Retention (how well you keep your users)

6.1%
Retention D1
47%
Retention D1
25%
Retention D7
6.2%
Retention D90
Compare with
6.1% D1 Retention
in the previous example

Example:
LTV ((Customer) Life Time Value)

User profitability is basically determined by LTV* and CAC*.
Being able to even roughly predict LTV can be very insightful.
KPIs
DAU
Monthly rev.
Retention D1 43 %
Retention D7 24 %
Retention D14 19 %
Retention D30 12 %
Conversion
ARPDAU
ARPPDAU
LTV
CAC
ARPDAU nn € nn €
Extr. tot. days (≤120th) nn nn
LTV nn € nn €
* LTV = (Customer) Life Time Value ; CAC = Customer Acquisition Cost

Example:
Liigapörssi (KPIs and tools)

Direct
Org. search
Email
SEM
Display
Other
Our main time unit of comparable comparison is a game
“phase”. The key desired user behavior varies within the phase
F
r
e
e
Trades
1
Pool
F
r
e
e
Trades
2
Pool
F
r
e
e
Trades
6
Pool
Direct
Org. search
Email
SEM
Display
Other
Pay
(Revenue)
Join pool
(Activation)
Sign up
(Acquisition)
Play the
whole phase
(Retention)
Continue to
next phase
(Retention)
Sign up or
reactivate
(Acquisition)
Desired user
behavior
Create pool
& invite
(Referral)
Invite friends
(Referral)
Make trades
(Activation)
Buy trades
(Revenue)
Create team
(Acquisition)
Phase 1 Phase 2 Phase 3-6
Season
Acquisition
Activation
Retention
Revenue
Referral
Acquisition
Referral
Revenue
Retention
Activation Focus area

We use various tools to identify development opportunities, run
tests and measure impact on the main high-level KPIs set
Area KPI
Acquisition
Signed up (%)
Created team (%)
CAC (€)
Activation
Active user (%)
In pool (%)
Retention
In-phase retention (%)
Phase change retention (%)
Revenue
LTV
ARPPU
Conversion
Referral Invited friend (%)
High-level (growth-stage-specific) KPIs to
continuously optimize
Tactical tools do explore user behavior and to
measure impact from change on KPI(s)
… …
Funnels
Emails
Anything!
Custom Dimensions
using hashed user
IDs enables user
level analysis, which
is VERY powerful

Some tools we are currently using
Tool Domain Sought benefits
Email Improve open rate and CTR
Email (transactional) Identify issues in use and funnels
Funnels Improve user onboarding and payments
Ecommerce Research revenue by product, user type, etc. (Enh. Ecom. Trac.)
Engagement Research anything at the user level!
Engagement Retention, User level progression
User Acquisition Volume and CPI (Cost Per Install)
Landing pages Optimize CTA (Call To Action) conversion
UI heat maps Research behavior and improve UI
KPI dashboard Communication and awareness

Some suggested further reading
* http://www.slideshare.net/dmc500hats/startup-metrics-for-pirates-long-version
Slideshare*

Building a data-driven culture

Building a data-driven culture

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (20)

Similaire à Building a data-driven culture

Similaire à Building a data-driven culture (20)

Dernier

Dernier (20)

Building a data-driven culture

Notes de l'éditeur