Research Operations and Scaling User Insights
•
11 j'aime•20,268 vues
Christian Rohrer: "Research Operations at Scale" DesignOps Summit 2017 • November 6-7, 2017 • Queens, NY, USA http://designopssummit.com
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (10)
Similaire à Research Operations and Scaling User Insights
Similaire à Research Operations and Scaling User Insights (20)
Plus de Rosenfeld Media
Plus de Rosenfeld Media (20)
Dernier
Dernier (20)
Research Operations and Scaling User Insights
- 1. RE SEARCH OPERATIONS AT S CALE: ENABLING OT HERS WHILE MAINTAINI NG EXCELLENCE C H R I S T I A N P. R O H R E R , P H D @christianrohrer and crohrer@yahoo.com https://www.linkedin.com/in/crohrer/
- 2. S O … W H AT C O M E S T O M I N D ?
- 3. KNOWN AS A CREDIT CARD COMPANY
- 4. BUT IT’S ACTUALLY A BANK…
- 5. G O A L : A T E C H C O M PA N Y ( T H AT J U S T H A P P E N S T O D O B A N K I N G ) Desirability ViabilityFeasibility
- 6. T O P S T R AT E G Y: H U M A N - C E N T E R E D D E S I G N Desirability ViabilityFeasibility (AND This means RESEARCH!)
- 8. DESIGN RESEARCH AT A BANK IS MIRED BY ”GOO” Research Goo Risk: Strategy Leaks Risk: Privacy Laws Risk: Compliance Reg: File 1099s Reg: Consumer Protection Reg: Complaints Inefficiency: Recruiting & Scheduling Inefficiency: Technical/Tool Complexity
- 9. AT C A P I TA L O N E , T H E R E A R E D E D I C AT E D R O L E S F O R R E S E A R C H O P E R AT I O N S A N D C U R R I C U L A Research Operations • Research Operations Leads: Research request intake, coordinate studies, manage recruitment vendors, scheduling participants, communicating research schedules • Technicians: manage lab and field equipment, videography Research Curricula • Instructors: Develop and teach courses on research methods & procedures to non-research employees • Specialists: Provide remote research tool licenses, access and training (e.g., survey tools, online studies, diary apps, etc.)
- 10. YOU CAN RUN A STUDY LIKE AN EVENT THIS RESU LTS IN 3 MAJOR BENEFITS RISK & REGULATION ARE HANDLED IN ADVANCE OR BY TRAINED PERSONNEL OTHER “GOO” LIKE RECRUITING, SCHEDULING AND TECHNICAL SETUP IS LARGELY REMOVED
- 11. EXAMPLE: “WINE AND DINE”
- 12. Should you do this?
- 13. CONTEXT
- 14. RESEARCH OPERATIONS MUST ADDRESS CHALLENGES FROM THREE KEY SOURCES Recent Trends in Research The Context of your business or industry Universal Challenges inherent in research
- 15. What about you? IT/INTERNAL B2BB2C Major distinctions: • Customers’ needs • Users’ choices • Customer/User Access
- 16. Or maybe you’re a startup. Who are your users today? What about tomorrow’s users? Are you getting direct access to your users? Diffusion of innovation
- 17. © 2015 Christian Rohrer WHAT PEOPLE SAY WHAT PEOPLE DO WHY & HOW TO FIX HOW MANY & HOW MUCH QUALITATIVE (DIRECT) QUANTITATIVE (INDIRECT) ATTITUDINAL BEHAVIORAL You really should be doing both qualitative and quantitative research
- 18. RESEARCH OPERATIONS MUST ADDRESS THE KEY SOURCES OF POTENTIAL CHALLENGES Recent Trends in Research The Context of your business or industry Universal Challenges inherent in research
- 20. AGILE IS HERE (DEAL WITH IT) • Why? Agile is a must to attract engineers • So UCD must fit Agile • By default, everything goes into a sprint • Everyone’s on the same team (pod) • Code early, no docs Source: http://scrumology.com/an-excerpt-from-improving-agility-agile-certifications/
- 21. Source: T O P R E S E A R C H C H A L L E N G E S W I T H T R A D I T I O N A L A G I L E • No UX roles defined • Little up front research • Hard to fit research into a sprint • Need to feed development • Dev centricity
- 22. R E C O N C I L I N G P R O C E S S E S Source: Peter Source: IDEO Source: Michael James & Luke Walter Developer & Product Owner Centric Designer & User CentricNote: all of these are “human-centered” (just for very different humans)
- 23. WORKING WITH AGILE – 2 MAIN APPROACHES • Approach 1: Use agile methodology, but in a separate design phase • Approach 2: Use an equally attractive, but more user-centered compatible processes (e.g., Lean) • IMO: Doing strict agile, in which UX is simply added to Dev Pods does NOT work for Human-Centered Design processes. You cannot design or research anything significant in the same sprint as development.
- 24. 24 EXAMPLE FROM INTEL SECURITY’S TRUEKEY DESIGN PROCESS (APPLYING AGILE TO EARLIER DESIGN PHASES) 2 weeks CXD sprint2-3 days before sprint starts What is relevant here: - Uses the same agile tools and nomenclature - Design happens BEFORE Development - Validate/Optimize designs/flows before code (on every sprint) - Research is scheduled in the ”design sprints” before design executes Dev
- 25. L E A N A P P R O A C H : B U I L D - M E A S U R E - L E A R N BUILD MEASURELEARN Products (MVP) UsageUptake
- 26. L E A N A P P R O A C H : B U I L D - M E A S U R E - L E A R N BUILD MEASURELEARN Products (MVP) UsageUptake Improvements Viable/Desirable Usability Interest Prototypes Ideas & Concepts
- 27. M O D I F I E D L E A N A P P R O A C H : B U I L D - S T U D Y- L E A R N BUILD MEASURELEARN Products (MVP) UsageUptake Improvements Viable/Desirable Usability Interest Prototypes Ideas & Concepts STUDY
- 28. Source: It’s our Research by Tomer Sharon Today, researchers can’t be the only ones driving research, or it would be a huge… Another Trend: Research Participation
- 29. DIY RESEARCH • Over the past 5-6 years, everyone wants to do research: • All design functions • Product managers • Developers • Even executives! • Removing barriers to research now essential. • Just enough training and support are needed
- 30. DIY RESEARCH TIPS • Get the right vendors for you • Reach the right user tyoe • Remove goo • Get tools that work • Provide training and support • Incorporate into YOUR process
- 31. RESEARCH OPERATIONS MUST ADDRESS THE KEY SOURCES OF POTENTIAL CHALLENGES Recent Trends in Research The Context of your business or industry Universal Challenges inherent in research
- 33. The 5 Purposes of Research 1. Understand people and problems 2. Inspire creativity and innovation 3. Validate concepts and solutions 4. Improve and inform designs 5. Assess and measure experience News flash! These require different skill sets, not always found in one person.
- 34. THE TALENT POOL: THERE ARE MANY SKILLS AND TALENTS • It’s hard to find people that can “do it all” • Not everybody wants to • You should allow for BOTH specialists and hybrid players on your team • This enlarges the talent pool
- 35. THE TALENT POOL: SMALL, GROWS SLOWLY, VICIOUSLY FOUGHT OVER “We will always be outnumbered.” -Whitney Quissenberry, panel member at UXPA2016, “UX Past, Present & Future” • Universities that produce Engineers: 1000s • Universities that produce UX Professionals: about 75 Conclusion: Don’t waste your design and research talent. Make them as efficient and strategic as you can. Source: indeed.com
- 36. W H AT K I N D O F R E S E A R C H W O R K S B E S T F O R D E S I G N E R S ? W H Y ?
- 37. WHAT KIND OF RESEARCH AND INSIGHTS WORK BEST FOR EXECUTIVES AND PRODUCT PROFESSIONALS? WHY
- 38. MANY THINK SIMPLISTICALLY: “BIG NUMBERS GOOD. SMALL NUMBERS BAD.” “Sorry, but I’ll never make a decision based on research with just 5 people.” –Product Management
- 39. OK, LET’S JUST ACKNOWLEDGE THAT METRICS MATTER 39 • “You can’t manage what you can’t measure” • “I need a dashboard to control my business” • “How does our NPS compare?” • “Invest in data scientists and big data” • “I want to be more scientific” • “Build, measure, learn” • “The design needs to be validated” Conversion rates, downloads, NPS, customer reviews, A/B testing performance, uptime, abandon rate, analytics data, path/click flows NOT measuring the experience. They measure the product performance.
- 40. A User-Centered Model of User Experience Look & FeelThe visual and industrial design is clear, professional and appropriate Sound Clear and relevant language, content and information architecture. The price of entry. In today’s world of technology, data, and design, there is no excuse for something to be hard. Ease Emotional Outcome It doesn’t matter how easy it is if it doesn’t address the real human need in the right way. Effectiveness Sources: • Rohrer’s Simple Model of UX (2006-2016) • Forrester: Three Es of CX (2015) 40
- 41. The PURE Method Pragmatic Usability Rating by Experts Source: Link goes here
- 42. 42 PURE: A way to measure “friction” PURE provides a cognitive load “score” based on : 1. A well-defined user type 2. A small set of “fundamental tasks” 3. The “performance” of those tasks (e.g., the Happy Path) for that user type, based on heuristics & design principles Like in golf, smaller numbers are better, and green is good.
- 43. In PURE, we judge the experience based on a ”rubric,” as in skating & gymnastics • A panel of raters each silently rates an experience flow they are all witnessing at the same time • A 1-3 rubric defines how much “friction” there is, based on known UX heuristics • PURE rates every step of the task (like a “move”) • Rather than averaging scores, the panel decides on the score of each step Photo credit: Ezra Shaw/Getty Images
- 44. A DIFFERENT KIND OF METRIC – ONE FROM A PANEL OF “EXPERTS” CONSIDER AN ANALOGY FROM THE REALM OF MOVIES… THE TOMATOMETER PURE is kind of like this: An score based on expert analysis
- 45. ACTUAL PURE SCORECARD OVER TIME (VERSION OVER VERSION) 45
- 46. ACTUAL COMPETITIVE PURE SCORECARD 46
- 47. E X E C S WA N T N U M B E R S … S O G I V E T H E M N U M B E R S ! H E R E ’ S W H AT H A P P E N S W I T H P U R E • People are intrigued. Then, either: o Owners and Designers of the product want to fix ease of use problems o Owners and Designers of the product push back, refuting PURE • Either outcome is good! We are finally focusing on improving an important aspect of user experience: Ease of Use. • It’s better to have numbers from actual user studies, but if you can’t do it easily, this is acceptable. It will cause positive change in your organization.
- 48. K N O W L E D G E M A N A G E M E N T I S H A R D
- 49. R E Q U I R E M E N T S O F A G O O D K N O W L E D G E M A N A G E M E N T S Y S T E M • Easy to find what we have and know • Easy to contribute new info and docs • Must handle more than text docs (shared cognitive artifacts like journey maps) • Tags and metadata • Include social and behavioral components (who contributed, who knows more, who used what, who likes what) • A place where knowledge can grow and be used, not just be stored and accessed
- 50. A J O U R N E Y M A P O F K N O W L E D G E Idea Question Research Finding Finding Counter Finding Counter Finding Better Idea Another Test Conclusion Fail Hypothesis Pivot Finding THE Idea Winner! Usability Study Find & Fix Find & Fix Find & Fix Find & Fix Find & Fix Find & Fix Build & Release Market data Success!
- 51. T H E P R O B L E M S PA C E I S L A R G E • Remove risk, deal with regulation, and address other industry-specific challenges • Teach/empower non-researchers • Find a way too fit into agile • Support qualitative research • Provide quantitative data • Keep track of and better use your insights across your design process
- 52. T H E P U R P O S E O F C A P I TA L O N E R E S E A R C H We believe opportunity lies in the understanding of the complexity of people. DEEP EXPERTISE INTERPRETATON EMPOWERMENTPARTNERSHIP Ultimately, we help bring insights into design.
- 53. @christianrohrer and crohrer@yahoo.com https://www.linkedin.com/in/crohrer/ For more on PURE, contact me or use #puremethod Thank you