Testing is an important part of a successful website as it allows for making changes on the fly that can lead to a much larger impact on your business. This section will outline the best practices for testing & personalization and our core methodology combining reach & impact to score the "best" test.

1. The Best Test:
Optimizing for Success

2. Presenter
Optimization Practice Lead,
Stratigent
TIM WALKER

3. This session will cover…
Why You Need to Test
To Test or Not to Test3
1
Testing Basics2
Bridging Testing to Personalization4

4. Why You Need To Test

5. Goal of testing
Finding the optimal customer experience…
Across medium (Web, app)
Across the spectrum of personas

6. Testing can help…
Perfect website design and flow
Maximize conversions
Provide insight into user behavior
Feedback on new features

7. Testing stats
Source: Forrester’s Q3 2015 Global Online Testing Platform Customer Online Survey

8. Pre-testing considerations
People
How much of an effect will
this test have on the users?
Process
Do you have the right
technology and skills in place
to develop and execute the
test?
Technology
Do you have the right
technology and skills in place
to develop and execute the
test?

9. Testing Basics

10. Testing Process
1.
Ideation &
prioritization
3.
Execution
2.
Implementation
4.
Analysis
5.
Iterate

11. Types of Tests
A/B test
Multivariate test
Multi-page tests
split up site traffic in a balanced
way, and then show one t of
users version A and the second
version B
setup to test many different
individual changes on a page
all at once
These tests involve changing
the experience across multiple
pages
CHANNELS
low high
C
O
M
P
L
E
X
I
T
Y
high

12. Client Example |
A/B Testing
Client was concerned that not having a
clearly defined 'guest' sign in option
for the checkout flow was driving away
customers from converting

13. The new design led to a higher
conversion level
Client Example |
A/B Testing
Tested different versions comparing
the default with a new design that had
guest sign-in front and center

14. Client Example |
Multivariate Testing

15. Client Example |
Multivariate Testing

16. To Test or Not to Test

17. Define what metrics to determine test
success
Translate a test idea into a formal hypothesis
Define what metrics to determine test success
Grade each hypothesis

18. Three dimensional grading
Reach
Impact
Technical difficulty
How many users will see this test?
How much of an effect will this test have on the users?
Do you have the right technology and skills in place to develop and
execute the test?

19. Metric selection
Metric selection is key to gauging test results, ensure
metrics that align with test goals & can detect
anomalies are selected.
Conversions | the primary metric of a test - measures
what you are trying to accomplish (e.g. orders, leads,
sign-ups)
Conversion
Secondary
Interactions
Tactical Engagement
Secondary Interactions | measure intermediate steps
towards conversion or post conversion (e.g. cart
additions, lead quality)
Tactical Engagement | these metrics monitor conversion
related values (e.g. units per order, average order value,
revenue, profit)

20. When not to test.
Your site design isn’t stable
Don’t know what they are trying to achieve
Don’t have the resources aligned in order to run a testing program
Aren’t mature enough to execute a complex test

21. Bridging Testing to Personalization

22. Testing as a precursor to personalization
We don’t have to treat all users the same.
If the retailor could predict the price the customers were willing to pay, then lower friction and high
conversion value. An optimization program could have produced $75 in revenue versus $50
User Example: Coupons for everyone
Same shirt, but two customers are willing to pay different prices…
A retailer generally releases
coupons for 50% off retail
price
Customer B: $25
Customer A: $50 (retail price)
Customer B pays $25
Customer A pays $25

23. Personalization modeling
Rules based
personalization
Build out individual
personas
Personalization based
on visit history
Personalization based on
integration of online/offline data
Customized Personalization Predictive Personalization
Using 1st hit
attributes
Triggers can change
experience on the fly
Customers fit into
audiences
Customers fit into behavioral
segments
Advanced targeting
Look-a-like Models
Segments fit onto
customers
Machine learning

24. Client Example |
A/B Testing
Client needed to test the size of tiles when page
viewed on mobile phones
Saw a 36% increase in click-thru rate with
wider tile format
A test was run on both layouts

27. THANK YOU
LinkedIn.com/Ebiquity
@ebiquityglobal
www.ebiquity.com