Last year I got a new job and spent the year running all the tests we've done on DistilledODN (an SEO split testing platform).
It's changed my perspective, taught me a huge amount and I'd like to take people through all the different lessons I've learned (19 of them in fact).
That's everything from: What sort of effect do basic SEO changes? Why is changing your title tags possibly a really risky move? How and when has structured data helped? How important is freshness (and can you fake it)? Does testing change your relationship with a client? Should you put emoji's in everything...
26. Cats Dogs Unicorns Badgers
CRO test - Different users, see different templates on the same pages
1
27. Cats Dogs Unicorns Badgers
CRO test - Different users, see different templates on the same pages
1
2
Cats BadgersDogs Unicorns
28. Cats Dogs
SEO test - different users see the same templates on different pages
1
2
Cats Dogs
Cats Dogs
29. Cats Dogs Unicorns Badgers
SEO test - different users see the same templates on different pages
1
2
Cats Dogs Unicorns Badgers
Cats Dogs Unicorns Badgers
30. Then we measure the change in traffic, e.g. 3% daily increase
60. How much traffic do you
need for SEO split testing?
Rule of thumb: you need roughly 1000 organic sessions a day to
the group of pages you’re testing.
@dom_woodman
103. Title tests usually change
traffic between ~5 - 15%
This stayed equivalent across site size and industry.
@dom_woodman
104. 56% of all title tag
changes were negative
Around 37% were null and only 6% were actually positive. It’s really
hard to write a good title.
@dom_woodman
106. You can’t stop testing
titles.
Titles exist in the context of the rest of a SERP, when everyone
copies your good title format (and they will) you’ll have to mix it
up again.
@dom_woodman
107. Title/meta effects typically
appear in 2-4 days.
It’s also easy to validate the changes have been picked up
because you can scrape it from the SERPS.
@dom_woodman
121. The same changes have
different effects on
different sites.
This is the big one. There really isn’t best practice.
@dom_woodman
122. (Assuming it was topically similar). We’ve saved clients a lot of
money, by showing they could reduce content without an effect.
Reducing content on
non-article pages was
often null.
@dom_woodman
138. The same changes have
different effects on
different sites.
This is the big one. There really isn’t best practice.
@dom_woodman
139. The same changes have
different effects on
different sites.
This is the big one. There really isn’t best practice.
@dom_woodman
THIS TIME I
ACTUALLY
LEARNED IT
140. Structured data has an
effect outside of rich
snippets
Occasionally we got big 10-15% wins on important templates, it
varied wildly and was mostly null (never negative).
@dom_woodman
141. Periodically re-challenge
your beliefs
I was lucky to test a successful site first. In a different order, I
might’ve given up on my hypothesis, before finding the right site.
@dom_woodman
150. If you don’t have intent,
bells and whistles fail.
5 star rich snippets increased traffic by 16% on the right site with the
right intent. When the intent wasn’t there, they appeared, but did
nothing.
@dom_woodman
159. Agile testing can help
you take larger risks.
If you can measure and quickly roll out/roll back a test, you can try
things you might not normally feel comfortable doing.
@dom_woodman
169. You don’t need to argue,
when you can test.
When you have solid testing framework and can build things quickly
and easily, testing is easier than arguing and removes arguments
from a relationship.
@dom_woodman
174. Happiness isn’t just up
and to the right
Instead the focus turns to other metrics like how many tests are we
running, how can we improve the testing process.
@dom_woodman
175. A negative test you rolled
back is a bullet dodged
Negative tests feel like wasted time. Once you realise without
testing you probably would’ve rolled out, it feels a lot better.
@dom_woodman
176. You’re about to be
forced out of silos.
You’re going to need to coordinate and work tightly with product
and QA teams if you’re not already.
@dom_woodman
183. Making changes to sections of pages
on templates
● Making SEO changes with tag
manager
● Cloud flare edge workers
● DistilledODN
General useful posts on testing
frameworks & velocity
● Hypothesis framework
● Running a weekly growth meeting
Do it yourself
How does split testing work?
● How does SEO split testing work?
Examples
● Pinterest - Demystifying SEO with
experiments
● Etsy - SEO title tag testing
Measuring SEO split tests
● Google’s original causal impact
paper
● A DIY tool for measuring SEO split
tests
● A walkthrough of the R Causal
Impact library