Beta testing produces enormous amounts of information on your product. But how do you know you're collecting the right feedback and data? Here our research team shows you how to set targeted goals for your next beta test and what methods you can use to reach them. Click this link to access the on-demand webinar: https://www.centercode.com/webinar/2016/september/

1. 1
Setting the Right Goals for Your Beta

2. 2
Meet the Speaker
John S. Little
Director of Research @ Centercode
john@centercode.com

3. Agenda
1. What are goals?
2. How to set good goals for your beta test
3. How to maintain your goals throughout the beta phase
4. How to connect the insights from your test to your goals
5. Q&A
3

4. 4
Who is Centercode?
Services Fully managed tests and programs
Software Complete beta program management platform
Testers Access to 150k+ deeply profiled global candidates
Content Free tools and best practices
Customer validation is our core competency
We've been collecting customer feedback for tech companies since 2001

5. Goals
Introduction
5

6. Webinar Goals
Establish good goals for beta tests
Accomplish goals of the beta test
Leverage information to make a better product
6

7. 7
Goals Refresher

8. Business Definition
An observable and measurable end result
having one or more objectives to be achieved
within a more or less fixed timeframe.
http://www.businessdictionary.com/definition/goal.html#ixzz4KBQiUDi6
8

9. Goals for betas
“Goal is something that you want to achieve”
How am I going to do it? - Method
Why am I going to do it? - Return
Useful tools
● S.M.A.R.T.
● Definition of Done
● Commander’s intent
9

10. Quote
If one does not know to which port one is sailing, no wind is favorable.
– Seneca
10

11. 11
Good Goals

12. Beta Goal Types
1. Overall Test Goal
2. Feature Set Level(s)
3. Attributes
12

13. Overall Test Goals - Finding a Focus
What is the main goal of the beta test?
● Natural usage
● Coverage of features
● Opinion of features
● Partial, focused coverage
● Unique environments and use cases
13

14. Real vs. Complete
Natural usage Coverage of features
14

15. Feature Set Goals - Finding a Focus
Where do I find the areas of the product that need focus?
● Product/Market Requirements Document (PRD/MRD)
● Marketing Messaging
● Packaging Design
● Key Stakeholder Requests
15

16. Priorities for Constraints
16
Constraints
● Time
● Products
● Cost
● Tester Availability
Feature Sets
● Setup
● Documentation
● Website
● App
● Settings
● Out of box
● WiFi
● Feature 1 - Feature n
VS

17. Attributes
17
Overall Satisfaction
Perceived Performance
Ease of Use
Usefulness
Preference
Comparative
Self Reported / Attitudinal

18. Communicate Goals
Document, distribute, and follow-up
● Key Stakeholders
● Immediate team members
● Extended team members (build awareness)
● Testers (if appropriate)
18

19. Goals for Feature Sets
19
Bad Good
Collect feedback on feature Measure user’s satisfaction with ease of use
of feature
Measure usability of the product Evaluate customer acceptance of product
Measure the performance of the product Measure perceived performance while
initiating application
How long did it take to setup the product? Evaluate satisfaction with time to set up
Will they pay $150 for it? Conjoint analysis
After Scenario Questionnaire (ASQ)

20. 20
Example

21. Product Example
Internet of Things Light
● Mobile App
● Preset Colors
● Manual Color Selection
● Timer
● Alarm
● Auto-dimmer
21

22. Beta Goals
● Evaluate total customer experience
● Assess MVP readiness
● Roadmap validation
● Competitive analysis
● Identify bugs
● Real-world impact of known issues
● Regression testing
● Analyze real-world performance
● Test compatibility
● Study user experience
● Assess new user experience
● Test documentation
● Test support processes
● Collect mock reviews
● Increase product confidence
● Collect case studies
22

23. Objectives
Quality Customer Acceptance Natural Use
● Test compatibility
● Assess MVP readiness
● Competitive analysis
● Identify bugs
● Regression testing
● Analyze real-world
performance
● Evaluate total customer
experience
● Assess new user experience
● Collect mock reviews
● Increase confidence
● Real-world impact of known
issues
● Study user experience
● Test documentation
● Test support processes
● Collect case studies
● Roadmap validation
23

24. SMART Beta
SMART Criteria
Specific Customer acceptance of product
Measurable Quantifiable by survey(s) & severity of issues
Achievable Product state / Tester availability / Tool
Relative Feedback will go into this version or next
Timebound 2 weeks of test -> 2 weeks regression -> GM
24

25. Definition of Done
❏ All new feature in the product tested by customers
❏ Measurable feedback on the customer acceptance of the product
❏ All discovered bugs prioritized and sent to QA
❏ Tester suggestions provided to product management
❏ Support alerted of specific pain points that are not being addressed
25

26. Commander’s Intent
Complete a beta study of the customer acceptance of the product
in order to
understand and make critical improvements prior to release or
general improvements in the next revision
26

27. Feature Set vs. Constraints
Features
● Packaging
● Easy Setup
● Preset Colors
● Manual Color Selection
● Timer and Alarm
● Auto-dimmer
Research Attributes
● Ease of Use
● Stickiness
● Usefulness
● Feature Satisfaction
Constraints
Units = 32
Time = 2 Weeks
Tools = Centercode
Testers = Betabound
Test Team = ½ Time (me)
27

28. Feature Set vs. Constraints
Features
● Packaging
● Easy Setup
● Preset Colors
● Manual Color Selection
● Timer and Alarm
● Auto-dimmer
Research Attributes
● Ease of Use
● Stickiness
● Usefulness
● Feature Satisfaction
Constraints
Units = 32
Time = 2 Weeks
Tools = Centercode
Testers = Betabound
Test Team = ½ Time (me)
28

29. Establish Goals
Primary Test Goal = Customer Acceptance
Feature Sets = Easy Setup; Preset Colors; Manual Color
Selection; Timer; Alarm
Attributes/Feedback
● Satisfaction, Ease of use, etc. (Surveys)
● Bugs (Issue reports)
Week 1
● Setup; Manual Colors; Timer
Week 2
● Alarm; Preset Colors; Timer
29

30. Centercode Test Plan
30

31. One more time
1. Identify test objectives
2. Group and provide reasonable options
3. Ensured that the goal is well crafted
4. Identified feature set goals & attributes
5. Prioritized based on constraints
6. Document and send out for review
Start
Finish
31

32. 32
Maintaining Goals

33. Surveys
Goal: Measure Attributes
Pros
Pointer to areas that might have problems
Very simple to send to large numbers of people
Cons
Qualitative aspects difficult to analyze
Attitudinal satisfaction with attributes of items
33

34. Bugs and Suggestions
Goals: Unique Environments and Use Cases
Pros
Facts & opinions from testers
Build both - let it sort for you
Cons
Difficult to quantify without tools
Can require back-and-forth for clarity
34

35. Journals
Goal: Natural Use Information
Pros
Diary type study
Requires limited guidance to testers
Cons
Requires more time for analysis
Could be limited
35

36. 36
Goal Dangers

37. Type Mismatch
Article - https://www.nngroup.com/articles/which-ux-research-methods/
Study Type Results
Attitudinal vs. Behavioral What they say vs. what they do
Qualitative vs. Quantitative Why/how to fix vs. how many / how much
37

38. Miscommunication
Goals not agreed upon
Feedback not solicited
New or unaware stakeholders
Misguided testers
Wrong methodology
38
¯_(ツ)_/¯

39. 39
Goals to Insight

40. Connect Feedback Types
Surveys and Bugs
Bugs and Suggestions
Surveys and Suggestions
Journals and …
Marketing, UX, Support, Sales, QA
40

41. Focus on Product Goals
Focus results on original goals - they were goals
for a reason
Bring in updates to the information that helped
you define the goals
Only highlight other areas if they are significant
41

42. Generate Top X List
Information overload happens to everyone
Prioritize for your own sake
Short list will get more attention than long list
Bring in the “good/promotes” from the test
Everyone likes to have a couple wins, and so does the customer
42

43. Problem Elevator Pitch
20 - 30 second rundown
Customer’s point of view
Focused on selling the item to the team
One for each item on “Top X” list
43

44. 44
Key Takeaways

45. If you remember nothing else...
1. Clearly define your goals - always; and associate
the best method to achieve them before signing off
on the project.
2. Stay focused on goals, unless significant change
3. Beware of research myopia
4. If you could use help, contact us!
45

46. 46
Q&A
John S. Little
Director of Research @ Centercode
john@centercode.com
Sign up for our newsletter and our
upcoming webinars
For more resources, visit our library

47. 47
Thank You!