The best way to improve products is to have people use them, but UX researchers struggle to share what they’ve learned in a way that has immediate and long-lasting impact. How do we keep the design process moving while grounding it thoroughly in research? This talk presents evidence for and against reports, and explores characteristics of reports that make them more and less successful at effecting change. We describe where approaches like debriefs, co-design, and video have succeeded and fallen short. Based on survey data from UX practitioners and experiences in the field, we address these questions: Is it worth it to write a report? Are there quicker, more engaging alternatives? What makes a compelling report? How do we make usability research usable? We offer a framework for choosing the best reporting approach, and share best practices for determining what to communicate, and how.
Kala jadu for love marriage | Real amil baba | Famous amil baba | kala jadu n...
The Report is Dead, Long Live the Report ! Communicating Usability Research Findings for Maximum Impact
1. The Report is Dead, Long Live the Report!
How to Communicate Usability Research Findings
for Maximum Impact
Kathi Kaiser
Centralis
@kathikaiser
www.linkedin.com/in/kathikaiser
2. • Co-founder of Centralis
• 20+ years making products easier, mostly
through user research
• Lots of training/mentoring of new UXers
along the way
• Observer and creator of various reporting
approaches
Hello!
Hi! I’m Kathi Kaiser.
3. Back in the day…
When I began in UX in the late 1990s, few people knew what UX research was.
5. And so…
Given our range of approaches, we should think critically about what we do, and why.
• How do we share research results?
• What seems to work best?
• What challenges are we facing?
• How should we adapt to different contexts?
• How can we get beyond “It depends…”?
7. The greater the distance between the
researcher and the stakeholders,
the more formal results sharing tends to be.
And we learned...
1. Distance demands
documentation.
8. Gaining buy-in for UX research results
is more about a collaborative process
than the artifacts produced.
And we learned...
2. Process
outperforms
artifacts.
9. The best way to package results depends
on the researcher’s available time and the
stakeholders’ available attention.
And we learned...
3. Optimize time
and attention.
13. Employee
Vendor
Not on the
Product Team
On the
Product Team
Distance Demands Documentation
Product
Team
Execs
Product
Owner
Designers
Devs
Where do UX researchers fit in?
14. Where do UX researchers fit in?
The Integrated Model (Employee)
Product
Team
Execs
Product
Owner
Designers
Devs
Product
Team
Execs
Product
Owner
Embedded UX
Researcher
Designers
Devs
Employee
Vendor
Not on the
Product Team
On the
Product Team
Distance Demands Documentation
15. Product Execs
Product
Owner
Designers
Devs
Employee
Vendor
Not on the
Product Team
On the
Product Team
Product
Execs
Product
Owner
Embedded
UX
Researcher
Designers
Devs
Distance Demands Documentation
The Integrated Model (Contractor)
Product
Team
Execs
Embedded UX
Researcher
Product
Owner
Designers
Devs
20. Distance Demands Documentation
Scope of Research Question
Range of Audience
Text vs. Visual
“I'll write up, here's what I think the
main findings were in basically a one-
page kind of thing.”
Time to Create
Shelf Life
“Everybody knows that producing
written documents in most
organizational cultures is not the
way to affect change.”
“Usually, we're bumping up at the
time when the product manager
has to hand [the product] over to
engineering for development.”
Sharing in the
Integrated Model
“It's mostly just the bare bones: what
did we observe, what did we learn
from it, what do we need to do now.”
21. Distance Demands Documentation
Scope of Research Question
Range of Audience
Text vs. Visual
“I sit within our Consumer Insights
Department, which is an overarching
research center of excellence – that
area supports the entire enterprise.”
Time to Create
Shelf Life
“There's like 15 years of
research…some of the insights,
drivers, motivations, those things
are still the same. And so they're
still relevant. And so it's helpful.”
Sharing in the
Service Model
“The way we write our reports is
having that executive summary up
front, because we are thinking
about the executives… tell me
what you're going to do. What
needs to be fixed and why?
22. Distance Demands Documentation
Scope of Research Question
Range of Audience
Text vs. Visual
A: “Maybe less than 20 hours for a
straightforward usability study…a
big user research project, like could
be like 40 or 50 or more hours.”
Time to Create
Shelf Life
“I always walk through the whole
report. So usually I book an hour,
which is not always enough.”
“I had a client reach out to me a
couple of months ago who was
like, we found your report from
three years ago and it's
incredible…we'd like you to do a
new project with us.”
Sharing in the
Outsourced Model
Q: “How long does it take you to write
a usability report?”
A: “Oh, forever. It takes forever.”
23. Distance Demands Documentation
Scope of Research Question
Range of Audience
Text vs. Visual
Time to Create
Shelf Life
Sharing in the
Outsourced Model
Scope of Research Question
Range of Audience
Text vs. Visual
Time to Create
Shelf Life
Sharing in the
Service Model
Scope of Research Question
Range of Audience
Text vs. Visual
Time to Create
Shelf Life
Sharing in the
Integrated Model
24. Distance Demands Documentation
“You want your stakeholders to DO something with your report; otherwise, your
report is useless. Why are you doing this work? It’s because you want change to
happen. The way to make change happen is to involve the stakeholders from
the beginning.”
26. “Making the results easy
to comprehend and
showing visual examples”
“Including an exec
summary of key insights/
recommendations so they
can quickly grasp the
"greatest hits".”
“Sharing participant
quotes/anecdotes from
testing”
“…video clips showing
participants failing (and
succeeding).”
Process Over Artifacts
Q: What have you found most effective for sharing usability findings?(2020)
Q: What have you found most effective for getting buy-in to the results of usability testing?(2023)
“Building relationships
and asking questions
about the plans and
roadmap”
“Require that
stakeholders observe
usability testing.”
“Collaboration on the
goals and planning of the
test”
“Review the test plan with
PM and UX and
Engineering key
stakeholders”
“Including responsible
stakeholders in planning
and analysis.”
“Having stakeholders
watch… take notes on a
shared sticky note space
and ‘affinitizing’ the
findings in a group
workshop.”
“Give stakeholders the
opportunity to
discuss/ask questions”
“Cross-functional,
collaborative analysis.”
“Socializing findings
ahead of the formal report
also helps, as do clear
recommendations.”
29. Process Over Artifacts
Level of Involvement: Integrated Model
R
Level
of
Involvement
R R R R
S S S S S
Integrated
Model
“I know the stakeholders I'm working
with and I know what problem we're
focused on at that time.”
30. Process Over Artifacts
Level of Involvement: Service Model
R
Level
of
Involvement
R R R R
S
S
S
S
S
“If we're going to do research for you,
you as a business owner need to
participate. You cannot just wait for
the report to be thrown over the wall.”
Service
Model
31. Process Over Artifacts
Level of Involvement: Outsourced Model
R
Level
of
Involvement
R R R R
S S
S
S
S
Outsourced
Model
“And then there's, you know, an analysis
phase…usually it's just me, where I try to
make sense of all the stuff that we saw in
the usability testing.”
32. Process Over Artifacts
Level of Involvement
R
Level
of
Involvement
R R R R
S S S S S
S
S
S
S
S
S S
S
S
S
Integrated
Model
Service
Model
Outsourced
Model
33. R
S
S
Process Over Artifacts
Boosting Stakeholder Involvement
Level
of
Involvement
R R R
S S S
S
S
S
S S
S
S
R
I S S
R
I S S
R
S
S
S
Integrated
Model
Service
Model
Outsourced
Model
34. R
S
S
Process Over Artifacts
Boosting Stakeholder Involvement
Level
of
Involvement
R R R
S S S
S
S
S
S S
S
S
R
I S S
R
I S S
R
S
S
S
Integrated
Model
Service
Model
Outsourced
Model
?
35. Process Over Artifacts
35
• The debrief session told us what
we need to know
• No one has time to write a report
• No one will read a report
• The design has already moved on
• The debrief is for the team;
the report is for the executives
• The report is our memory
• The report is our roadmap
• The report is our product
The Report is Dead! Long Live the Report!
Integrated
Model
Service
Model
Outsourced
Model
37. Optimizing for Time and Attention
A Taxonomy of Reports (Cooley & Barnum, 2019)
Cooley, D., Barnum, C. (2019). How Do Other People Do It? A Comparative Review of Usability Study Reports. User Experience Magazine, 19(1).
Retrieved from https://uxpamagazine.org/how-do-other-people-do-it-a-comparative-review-of-usability-study-reports/
Visuals from DialogDesign: https://www.dialogdesign.dk/cue-studies/
‘Traditional’ Reports ‘Visual’ Reports ‘Lean’ Reports
38. Optimizing for Time and Attention
Report Building Blocks
38
Key
Findings Product
Images
Quotes
Video
Clips
Data Viz
Detailed
Findings Severity
Ratings
Session
Metrics
Recos
Method
What we did How to illustrate it What to do about it
What we learned
39. Optimizing for Time and Attention
Report Building Blocks
39
Key
Findings
Detailed
Findings Severity
Ratings
Recos
Method
What we did How to illustrate it What to do about it
Traditional
Focus
What we learned
Product
Images
Quotes
Video
Clips
Data Viz
Session
Metrics
40. Optimizing for Time and Attention
Report Building Blocks
40
Key
Findings
Detailed
Findings Severity
Ratings
Recos
Method
What we did How to illustrate it What to do about it
Visual Focus
What we learned
Product
Images
Quotes
Video
Clips
Data Viz
Session
Metrics
41. Lean Focus
Optimizing for Time and Attention
Report Building Blocks
41
Key
Findings
Detailed
Findings Severity
Ratings
Recos
Method
What we did How to illustrate it What to do about it
Lean Focus
What we learned
Product
Images
Quotes
Video
Clips
Data Viz
Session
Metrics
42. Optimizing for Time and Attention
Report Building Blocks
Key
Findings
Product
Images
Quotes
Video
Clips
Data Viz
Detailed
Findings
Severity
Ratings
Session
Metrics
Recos
Method
Stakeholder
Attention
Researcher Time
43. Optimizing for Time and Attention
The Attention Paradox: An Exercise
43
N P R F D R C I A P D Q B M W J F K
44. Optimizing for Time and Attention
The Attention Paradox: An Exercise
44
N P R F D R C I A P D Q B M W J F K
45. Optimizing for Time and Attention
The Attention Paradox: An Exercise
45
N P R F D R C I A P D Q B M W J F K
N P R F D R C IA P D Q B M W J F K
46. Optimizing for Time and Attention
The Attention Paradox
46
Our attention is limited.
The less attention needed to process something,
the more attention available for processing it.
We capture more attention by requiring less attention.
47. Optimizing for Time and Attention
Report Building Blocks
Key
Findings
Product
Images
Quotes
Video
Clips
Data Viz
Detailed
Findings
Severity
Ratings
Session
Metrics
Recos
Method
Stakeholder
Attention
Researcher Time
48. Optimizing for Time and Attention
Report Building Blocks
Product
Images
Quotes
Video
Clips
Data Viz
Detailed
Findings
Severity
Ratings
Session
Metrics
Recos
Method
Stakeholder
Attention
Researcher Time
Key
Findings
49. Optimizing for Time and Attention
The Elevator Pitch Game
Claire Slattery, https://claireslatterycoaching.com/
50. Optimizing for Time and Attention
Report Building Blocks
Quotes
Video
Clips
Data Viz
Detailed
Findings
Severity
Ratings
Session
Metrics
Recos
Method
Stakeholder
Attention
Researcher Time
Key
Findings
Key
Findings
Product
Images
51. Optimizing for Time and Attention
Report Building Blocks
Video
Clips
Data Viz
Detailed
Findings
Severity
Ratings
Session
Metrics
Recos
Method
Stakeholder
Attention
Researcher Time
Key
Findings
Key
Findings
Product
Images
Key
Findings
Product
Images
Quotes
55. Optimizing for Time and Attention
Report Building Blocks
Detailed
Findings
Severity
Ratings
Session
Metrics
Recos
Method
Stakeholder
Attention
Researcher Time
Key
Findings
Key
Findings
Product
Images
Key
Findings
Product
Images
Quotes
Key
Findings
Product
Images
Quotes
Video
Clips
Key
Findings
Product
Images
Quotes
Video
Clips
Data Viz
56. Method
Method
Optimizing for Time and Attention
Report Building Blocks
Detailed
Findings
Severity
Ratings
Session
Metrics
Recos
Method
Stakeholder
Attention
Researcher Time
Key
Findings
Key
Findings
Product
Images
Key
Findings
Product
Images
Quotes
Key
Findings
Product
Images
Quotes
Video
Clips
Key
Findings
Product
Images
Quotes
Video
Clips
Data Viz
Severity
Ratings
Session
Metrics
Move to Appendix
Move to Debrief & Preview
Exclude
Severity
Ratings
57. Optimizing for Time and Attention
Report Building Blocks
Detailed
Findings
Recos
Stakeholder
Attention
Researcher Time
Key
Findings
Key
Findings
Product
Images
Key
Findings
Product
Images
Key
Findings
Product
Images
Key
Findings
Product
Images
Stakeholder
Attention
Researcher Time
Detailed
Findings
Detailed
Findings
Product
Images
Detailed
Findings
Product
Images
Quotes
Detailed
Findings
Product
Images
Quotes
Video
Clips
Detailed
Findings
Product
Images
Quotes
Video
Clips
Data Viz
Quotes Quotes
Video
Clips
Quotes
Video
Clips
Data Viz
Quotes Quotes
Video
Clips
Quotes
Video
Clips
Data Viz
58. Optimizing for Time and Attention
Report Building Blocks
58
Finding = Behavior Why did they do that?
How did they react?
What happened next?
+ Cause
+ Mindset
+ Impact
60. Optimizing for Time and Attention
Progressive Specificity
Navigation to Screen
Screen Layout
Individual Screen Elements
Call to Action
Next Screen >
61. Optimizing for Time and Attention
Should researchers make recommendations?
No Yes
Recos
62. Optimizing for Time and Attention
Should researchers make recommendations?
62
“That can create a silo effect and potentially, antagonism.
Research has come here to tell you all the things that are wrong
and all the things you should have done under the guise of
‘recommendations’.”
“You don't tell me what to do
and I won't tell you what to do
because obviously you're a
professional who has skills
and knowledge that I don’t.”
No Yes
63. Optimizing for Time and Attention
Should researchers make recommendations?
63
No Yes
“Some product managers were like, who do
you think you are, trying to drive these next
steps. That was when I was new. Now, I’m
like, hey, we're having next steps meeting
from the research and we're going to all
discuss together what we're doing next.”
64. Optimizing for Time and Attention
Should researchers make recommendations?
64
“Recommendations are a key part of what we do.
Why be the one who holds all of the knowledge on
the findings and not tell people what they should,
ideas for things they should do about that?”
No Yes
65. Optimizing for Time and Attention
Should researchers make recommendations?
65
“I usually just do it because I need it to
make the next steps in an iteration.”
No Yes
67. The greater the distance between the
researcher and the stakeholders,
the more formal results sharing tends to be.
Choose your reporting approach in part
based on whether you work in an
integrated, service, or outsourced model.
In Summary
1. Distance demands
documentation.
68. Gaining buy-in for UX research results
is more about a collaborative process
than the artifacts produced.
Loop in stakeholders early and often.
In Summary
2. Process
outperforms
artifacts.
69. The best way to package results depends
on the researcher’s available time and the
stakeholders’ available attention.
Choose your components wisely, and learn
how to lighten their load on everyone.
In Summary
3. Optimize time
and attention.