5. Top F***ups for 2014
1. Testing in the wrong place
2. Your hypothesis inputs are crap
3. No analytics integration
4. Your test will finish after you die
5. You don’t test for long enough
6. You peek before it’s ready
7. No QA for your split test
8. Opportunities are not prioritised
9. Testing cycles are too slow
10. You don’t know when tests are ready
11. Your test fails
12. The test is ‘about the same’
13. Test flips behaviour
14. Test keeps moving around
15. You run an A/A test and waste time
16. Nobody ‘feels’ the test
17. You forgot you were responsive
18. You forgot you had no traffic
19. You ran the wrong test type
20. You didn’t try all the flavours of testing
slidesha.re/1wBbZ9c
@OptimiseOrDie
14. The 95% Problem
• Many people use 95, 99% ‘confidence’ to stop
• This value is unreliable
• Read this Nature article : bit.ly/1dwk0if
• You can hit 95% early in a test
• If you stop, it could be a false positive
• Tools need to be smarter about inference
• This 95% thingy – it’s last on your list for
reasons to stop testing
@OptimiseOrDie
15. Business & Purchase Cycles
@OptimiseOrDie
Start Test Finish Avg Cycle
• Customers change
• Your traffic mix changes
• Markets, competitors
• Be aware of all the waves
• Always test whole cycles
• Minimum 2 cycles (wk/mo)
• Don’t exclude slower buyers
17. When to stop?
• Know the cycles
• Run a test length calculator - bit.ly/XqCxuu
• Set your test run time
• Yup, just like a retail store
• Run it
• Stop it
• Analyse the data
• That’s it. No peeking and waiting for the numbers
to look good so you can stop.
• Go again
@OptimiseOrDie
19. QA Test or Die!
• Over 40% of tests have had QA issues.
• It’s very easy to break or bias the testing
Browser testing www.crossbrowsertesting.com
www.browserstack.com
www.spoon.net
www.saucelabs.com
www.multibrowserviewer.com
Mobile devices www.appthwack.com
www.deviceanywhere.com
www.opendevicelab.com
Article bit.ly/1wBccsJ
@OptimiseOrDie
20. Gamble the Company
AWAY!
• I get 60-70% right
• UX and Copywriters good at picking!
• C level execs are easy marks
• Ironically, many decide ‘designs’
• You need collaborative test design
• It’s a team game, with customers
• Flip a coin, anyone?
27. 23 : Business Future Testing?
“Congratulations!
Today you’re the
lucky winner of our
random awards
programme. You
get all these extra
features for free,
on us. Enjoy.”
28. #1 Smart Talented Polymath People
The 5 Legged Optimisation
Barstool
@OptimiseOrDie
Flexible and Agile teams
33. #4 : GREAT COPYWRITING
“On the average, five times as many people
read the headline as read the body copy. When
you have written your headline, you have spent
eighty cents out of your dollar.”
David Ogilvy
“In 9 years and 40M split tests with visitors, the
majority of my testing success came from
playing with the words.”
@OptimiseOrDie
41. #12 : The Best Companies…
• Invest continually in analytics instrumentation, tools, people
• Use an Agile, iterative, cross-silo, one team project culture
• Prefer collaborative tools to having lots of meetings
• Prioritise development based on numbers and insight
• Practice real continuous product improvement, not SLEDD*
• Are fixing bugs, cruft, bad stuff as well as optimising
• Source photos and content that support persuasion and utility
• Have cross channel, cross device design, testing and QA
• Segment their data for valuable insights, every test or change
• Continually reduce cycle (iteration) time in their process
• Blend ‘long’ design, continuous improvement AND split tests
• Make optimisation the engine of change, not the slave of ego
* Single Large Expensive Doomed Developments