Quant + Qual + Iteration for Great Products

Combining Quantitative,
Qualitative, and Iterative
Methods for Great Products
Becoming Product Led Workshop Series

Becoming Product-Led2
Ben Carey
VP, Professional Services at Pendo
bencarey@pendo.io

Becoming Product-Led
THE PACE OF CHANGE
IS EXPONENTIAL
4

data from

SOFTWARE HAS EATEN
THE WORLD
8

data from

OPPORTUNITIES
ARE RICH
COMPETITION
IS FIERCE
11

QUANTITATIVE
+ QUALITATIVE
+ ITERATION
12

Quantitative research tells you
WHAT your problem is.
Qualitative research tells you
WHY you have that problem.
13
LAURA KLEIN

...and iteration allows you to ﬁx it.
14

Understanding how to act on data has the prerequisite that we
understand what the data is to begin with. The insights are
there, but you have to be able to interpret them to turn data
into information.
In addition to understanding deﬁnitions, nuances, and
benchmarks - you should also understand data and time-series
visualization methods to spot trends, track progress, ﬁnd
outliers, and fuel hunches.
➔ EV, EPS, OPEX, CAPEX, ROA, ROE, IRR, DCF
➔ ARR, CAC, LTV, ASP, MAUs/DAUs, NRR
➔ NPS, CSAT
➔ Retention, Page / Feature Usage, Views, Clicks, Events, Conversion
BECOME DATA LITERATE
Data and visualization are the language of good decision
making. Learn to use both to your advantage..

Analyze the few critical things that answer the right questions.
17
Analyze and watch the important things on a regular cadence.
To determine what those KPIs are - you have to first find the
right questions you need answered.
Start with the questions instead of with the measures. When you
start with the metrics - you’ll tend to track for trackings sake. Find
the metrics that answer your questions.
Different stages of companies and different types of products
will have different questions. Cohort definition and aggregations
of metrics can often be insightful.
As a whole, the company should be tracking to a handful of
critical company-wide KPIs. And each team or department can
go more detailed when / as needed.
ASK THE RIGHT QUESTIONS TO DEFINE KPIs

Most of us have been conditioned to guess and take shots in
the dark. This is an outdated mindset at best, and can easily
be company-destroying proposition.
Most of us have operated for years on leaps-of-faith that have
little grounding on real data and usage information.
With physical objects instrumentation was often impossible. With
traditional software instrumentation was diﬃcult and rarely
prioritized. In the world of SaaS - instrumentation can be
automatic and often requires only a few keystrokes.
TELEMETRY IS A NECESSITY
Telemetry in software is simply table stakes to make better
decisions for the evolution of your application..

80% of the surface area of our applications are rarely or never
used. This is the unfortunate reality of what we’re building and
it’s also the most-likely opportunity for improvement in our
industry.
Usage and feature adoption isn’t everything - but it is almost
everything. Yes, In some cases this is expected - but those are
the exceptions and not the rule.
Many of us have suspected this was the case and we were right.
The good news is that these are clear opportunities for
improvement and learning. And improvement and learning are
what leads to creating products that people love.
THE POWER OF PARETO
We spend time where we get value.

How many employees working on your most
recent release? PMs, UX, Devs, Testers, Etc.
How long did it take (months)?
employees * $12,000 * months * 0.24 … 0%
employees * $12,000 * months * 0.80 … 5%
ex: Avg dev fully loaded salary: $12k per month

It kinda feels
like this

24 If You Build it, Will They Come? Webinar
Feature
Adoption Matrix

Hidden Gems
HIGH SENTIMENT, LOW USAGE
● Promote within your customer base
● Likely your next diamonds

Diamonds
HIGH SENTIMENT, HIGH USAGE
● Congratulations! You nailed it.
● Now activate your advocates

Fool’s Gold
LOW SENTIMENT, LOW USAGE
● Sometimes a bet doesn’t pay oﬀ
● Consider these features for retirement

Diamonds in the Rough
LOW SENTIMENT, HIGH USAGE
● Good news: the feature has promise.
Bad news: it’s not delivering value
● Engage users to understand how it’s
missing the mark

Better retention = better acquisition, monetization, virality, and unit economics.
29
In B2B, it’s usually important. In B2C, it’s always important.
You have almost no time to get users engaged and delighted.
Changing the bend of a cohort-based retention curve is likely
your biggest opportunity.
Stabilization of the curve indicates that some users are getting
value out of your application. Early retention / churn will heavily
inﬂuence unit economics.
Bending the curve as early as possible will give you
disproportionate returns. Most churned users churn incredibly
quickly. Focus heavily on early experience and onboarding.
RETENTION IS (USUALLY) #1

B2B Pattern (Months)
30
B2C Pattern (Days)

Ad-Driven Subscription Transactional Freemium
More ads served Lifetime Value More volume More upgrades
RETENTION DRIVES MONEY THROUGH...

What can we do to address retention?

The dynamics of stickiness vary with diﬀerent types of products.
34
The Daily Active Users (DAU) to Monthly Active Users (MAU)
Ratio measures the stickiness of your product - that is, how
often people engage with your product. Variations with Weekly
Active Users (WAU) may also be useful.
DAU is the number of unique users who engage with your product in a
one-day window. WAU is the number of unique users who engage with your
product over a 7 day window (usually a rolling 7 days). MAU is the number of
unique users who engage with your product over a 30-day window (usually a
rolling 30 days).
The ratio of DAU to MAU is the proportion of monthly active users who
engage with your product in a one-day window.
PRODUCT STICKINESS (DAU / WAU / MAU)

LOYALTY BY APPLICATION CATEGORY
https://andrewchen.co/what-factors-inﬂuence-daumau-nature-versus-nuture/

Sentiment can be much more powerful as a dimension than as stand alone number.
36
Unique to B2B software, analyzing NPS at the
account-level can oﬀer a tremendous amount of
insight.
When combining product usage and ARR size, this
scatterplot quickly highlights accounts that may be at
risk. It should prompt immediate action if not yet
accounted for.
It’s also important to know and understand those
accounts that have not provided us NPS. It prompts the
need to ﬁnd alternative ways of understanding
sentiment--especially for the larger accounts.
ACCOUNT-LEVEL NPS

NPS segmented by persona can lead to great insights.
37
For user level NPS, look for the overall scores and
scores by persona.
Ultimately, you are building for a speciﬁc persona, so it’s
likely that score is higher than overall.
Reading each verbatim and pulling out themes is also
helpful when looking to drive improvements in the
score.
Remember, NPS measures the likelihood to recommend.
USER-LEVEL NPS

VISIBLE SENTIMENT AS A CULTURAL ANCHOR

What are some quant metrics that you
are currently tracking?
What are some quant metrics that
you’d like to track? Why aren’t you?

What has surprised you? What is an
example of something that was
drastically diﬀerent than what you
expected when you measured it?

Peeling back the layers can lead to higher planes of
awareness and you can have a lot of fun while you’re at it.
Unfortunately, these concepts are often ignored.
A variety of tools exist to guide your observation and evaluation
of the journey. Hold space for these activities and let openness
take you on the ride to deeper insights.
➔ Design Thinking
➔ Contextual Inquiry
➔ Human-centered Design
➔ Observation
➔ Ethnography
GO WAYFINDING
Having the discipline to detach and run the process. You’ll
be rewarded with innumerable insights to guide the future
direction of your product..

A healthy dose of human-centered design will go a long way
to solving authentic problems. You’ll need to dig beyond the
simple feature requests and the ubiquitous feature race. Go
deep and design solutions for the problems you’re observing.
Feedback is everywhere, and it can often be overwhelming.
Look for patterns beyond the surface-level statements. Be
curious by asking powerful questions. Test ideas on the ﬂy when
you’re in the ﬁeld and utilize in-app testing where possible. Don’t
let the loudest personalities and the deepest pockets win.
THEIR PROBLEM, YOUR SOLUTION
Simply building what people ask for is doing a disservice to
your customers, your company, and to yourself. You’re a
professional - recognize that.

We’re rarely sitting on a lack of feedback. All too often, there
are troves of good feedback and information hiding in plain
sight. Revisit these occasionally and look for patterns.
All of these sources are often ﬂoods of information and we often
turn these channels oﬀ so that we can focus on go deep.
Customer Success, Professional Services, Sales Engineers, NPS,
G2, public forums, tweets, and similar channels all exist. These
items should all be inputs for consideration. And often - these
can spark new ideas or approaches as you go deeper and
explore more intentionally.
DIG THROUGH THE CRATES
If someone cares enough to provide the feedback, your
team should take the time to consider it..

What are some qual tactics that you
are currently using?
What has led to the deepest insights?

A large opportunity exists in using Cost of Delay as a
first-class citizen in our ways of approaching product
development. Most of the time spent in any process is waiting,
and when that waiting time is cut out - an immense world of
possibility is opened.
We’ve been trained and conditioned to focus on efficiency by
evaluation time invested and effort. When we open up our
thinking by also considering outcome-attainment and value then
we’ll often come to the conclusion that value delivered sooner is
often better for our users as well as our organizations.
ACKNOWLEDGE THE COST OF DELAY
Remember the pareto findings? What would happen if we
delivered that 20% - and did it 50% faster?

Ask yourself “How can I make this easy?” - and then put it in
action. Pixel-perfect prototypes and traditional user testing are
almost always overkill. Directionally-correct and narrowing in
on your pool of options will lead you to the right place in the
majority of scenarios.
Consider alternative approaches to move more quickly.
➔ Paper prototyping
➔ In-app feedback
➔ Discount usability testing
GET CREATIVE.
Consider your return-on-time-invested. How can this be easy?.

We’ve spent our professional lives ﬁxated on getting
something “done” - but the reality is that we never reach that
magical ending point. We need to be comfortable with
building to learn.
You’ll never know as little as you do right now. Our software and
services will (and should) constantly evolve because no industry
or business or problem is sitting still. While it feels good to have
“a plan” - we need to acknowledge that plans are about the
future, and the future is uncertain. Our frame of reference on
building needs to be reoriented from building to build into one of
building to learn.
BUILDING TO LEARN
Failing fast isn’t the goal. Learning fast is.

Become data literate.
The right questions lead to the right KPIs.
Telemetry is a necessity.
The power of pareto.
Addressing retention.
Addressing stickiness.
Account-level NPS.
User-level NPS.
Go wayﬁnding.
Dig through the crates.
Their problem, your solution.
Acknowledge the cost of delay.
Get creative.
Build to learn.
A QUICK RECAP OF THE TACTICS
Quantitative
Qualitative
Iteration

QUANTITATIVE
+ QUALITATIVE
+ ITERATION
54

LIFE IS TOO SHORT TO
SPEND YOUR TIME
HERE BUILDING
MEDIOCRE SOFTWARE.
#leavealegacy
#productlove

Quant + Qual + Iteration for Great Products

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Quant + Qual + Iteration for Great Products

Similaire à Quant + Qual + Iteration for Great Products (20)

Plus de Ben Carey

Plus de Ben Carey (8)

Dernier

Dernier (20)

Quant + Qual + Iteration for Great Products