In this workshop we'll show you how to: - Set the right testing goals that drive results - Create a test hypothesis using our experiment hypothesis framework - Analyze your results and communicate the value of data to key stakeholders

1. Lessons from the World’s
Greatest Testers
May 2016
@Hazjier

2. Hazjier Pourkhalkhali
Lead Strategy Consultant
@Hazjier

3. Optimizely is the world’s largest digital laboratory
Alexa 10k websites, Q1 2016

5. We researched the world’s top companies
Interviewed experts
§ Spoke with thought
leaders in the industry
§ Met with top customers
Built maturity model
§ Developed
comprehensive
maturity model
§ Assessed 100
customers
§ Surveyed 50+
customers
Held 100+ Business Reviews
§ Met with executives
at over 100
Optimizely clients
Analyzed Top Revenue Experiments
§ Analyzed the 100 best
experiments run on
Revenue Per Visitor
§ Asked customers to
describe test and process

6. 1 Understand your
revenue levers
3 Solve real
problems
2 Test all changes
4 Test multiple
variations

7. 1 Understand your
revenue levers

8. User
acquisition:
1.5m
User
retention
33%
Add-to-Cart:
4.00%
Cart
Checkout:
50.0%
Average
quantity:
2.00
Average per
unit price:
£25.00
Average
Order Value:
£50.00
Conversion
Rate:
2.00%
Revenue Per
Visitor:
£1.00
Visitors:
2.0m
Revenue:
£2.0m
Understand the revenue drivers

9. Goals
Strategies
Tactics
Cart
Checkout
Rate
Wording /
messaging
Location /
size
Design /
color /
iconograp
hy
Establish a
visual
hierarchy
Remove
non-
critical
content
Limit
number of
choices
Select/
target the
right USP’s
Refine the
messaging
Crystallize
visually
Clear and
consistent
UI
Structure,
naming,
and order
of sections
Describe
visually
Test ideal
product
visuals
Use the
right
review
systems
Descriptions
Reviews
Images
Emphasize
the Primary
Call to Action
Minimize
Distractions
Communicate
Unique
Selling Points
Design
Intuitive
Navigation
Market
Products
Effectively
Build a Sense
of Urgency
Show
quantity
left
Push
temporary
offers
Promote
buying
before set
times

10. 2 Test all changes

11. “Every change we make to our
website has to be approved
by our senior staff”
-Financial Services company
Testing requires freedom
LOW FREEDOM HIGH FREEDOM

12. Testing requires freedom
LOW FREEDOM HIGH FREEDOM
“Our Brand Styleguide is key.
We cannot make changes that
do not fit our design book”
-Luxury goods company

13. Testing requires freedom
LOW FREEDOM HIGH FREEDOM
“We would even change
our logo if it converted
better”
-Travel Company

14. 3 Solve real
problems

16. NO BEARD CAVEMAN
.30% .29% .28% .32% .25% .74%
.11% .28% .24% .32% .09% .59%
.74%
.59%

17. Minimize Distractions:
Remove Non-Critical Content
+14%
Revenue Per
Visitor

18. BASED ON KEYWORDCURATED
+24%
Click to Sale
+30%
Add-to-Cart
+210%
Conversions

19. Problem Solution Result
All great hypotheses begin with the problem, NOT the
solution!

20. 4 Test multiple
variations

21. The problemwith “A/B Testing”
2
A B
4
A B C D

22. Most people test only 2 variations
0
10,000
20,000
30,000
40,000
50,000
60,000
2 3 4 5 6 7 >8
Tests
65% 17% 17%
# of Variations

23. Most people test only 2 variations
0
10,000
20,000
30,000
40,000
50,000
60,000
2 3 4 5 6 7 >8
Tests
# of Variations

24. Test 5 variationsand nearly double your win rate!
Win Rate
# of Variations
14% 18% 20% 27% 20% 22% 23%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2 3 4 5 6 7 >8
+90%

25. Running more variationswill significantlyimprove your testing!

26. 1 Understand your
revenue levers
3 Solve real
problems
2 Test all changes
4 Test multiple
variations

27. Workshop

28. § The <user persona>
§ Cannot find…
§ Does not know…
§ Does not see…
§ Wants to…
§ Is distracted by…
§ Is forced to…
§ Needs…
Problem Solution Result
§ By <…> the user
will… <…>
§ If <solution> solves
<problem>, then
<metric> will rise
Build powerful hypotheses

29. § Discover 5-10 problems for
your users
§ Group problems by themes
§ Prioritize most important
problems
Problem Solution Result
§ Select 1 concrete
problem
§ Design 10 solutions per
theme
§ Choose 4 preferred
solutions
§ Primary Goals:
What do you seek to impact
immediately
§ Secondary Goals:
What would you like to
impact, but do not have to
§ Monitoring Goals:
What are metrics you want
to track
Exercise: Build Powerful Hypotheses

30. Optimizely funnel analysis
Homepage
Flight
Results
Book
Flight
Passenger
Details
Flight
Options
Payment
Page
Step
45%
30%
65% 65%
90%
70%
40%
40%
15%
20%
5%
25%
15%
30%
20%
15%
5% 5%

31. Optimizely funnel analysis
Homepage
Flight
Results
Book
Flight
Passenger
Details
Flight
Options
Payment
Page
45%
30%
65% 65%
90%
70%
40%
40%
15%
20%
5%
25%
15%
30%
20%
15%
5% 5%
Step
Completion

32. Optimizely funnel analysis
Homepage
Flight
Results
Book
Flight
Passenger
Details
Flight
Options
Payment
Page
45%
30%
65% 65%
90%
70%
40%
40%
15%
20%
5%
25%
15%
30%
20%
15%
5% 5%
Step
Divergence

33. Optimizely funnel analysis
Homepage
Flight
Results
Book
Flight
Passenger
Details
Flight
Options
Payment
Page
45%
30%
65% 65%
90%
70%
40%
40%
15%
20%
5%
25%
15%
30%
20%
15%
5% 5%
Step
Exit Rate

34. Optimizely funnel analysis
Homepage
Flight
Results
Book
Flight
Passenger
Details
Flight
Options
Payment
Page
Step
45%
30%
65% 65%
90%
70%
40%
40%
15%
20%
5%
25%
15%
30%
20%
15%
5% 5%
Exit Rate
Divergence
Completion

35. Optimizely funnel analysis
Homepage
Flight
Results
Book
Flight
Passenger
Details
Flight
Options
Payment
Page
Step
Primary /
Secondary
Goals
§ Searches
Submitted
§ Category tabs
§ Deals
§ Frequent Flyer
§ Flight Selection
§ Filter Usage
§ Refine Search
§ Sort Order
§ Continue
button
§ Edit search
§ Upsells
§ Field
completion
rate
§ Continue
button
§ Continue
button
§ Upsells
§ Seat
selection
§ Conversion
rate
§ Form field
completion
§ Payment
Type
Monitoring
Goals
§ Checkboxes
§ Flight tracker
§ Home page
clicks
§ Menu
interactions
§ Site search
§ Home page
clicks
§ Menu
interactions
§ Site search
§ Field error
rate
§ Home page
clicks
§ Menu
interactions
§ Site search
§ Shopping
cart
pageviews
§ Home page
clicks
§ Return to
previous page
§ Edit address,
information
§ Error rate

36. § Discover 5-10 problems for
your users
§ Group problems by themes
§ Prioritize most important
problems
Problem Solution Result
§ Select 1 concrete
problem
§ Design 10 solutions per
theme
§ Choose 4 preferred
solutions
§ Primary Goals:
What do you seek to impact
immediately
§ Secondary Goals:
What would you like to
impact, but do not have to
§ Monitoring Goals:
What are metrics you want
to track
Exercise: Build Powerful Hypotheses