A view on pre-testing research by James Hurman, Head of Planning at Colenso BBDO and author of The Case for Creativity

1. The case for!
moderation!
!
A view on pre-testing research!
!
!
James Hurman!
Head of Planning, Colenso BBDO, New Zealand!
Author of The Case for Creativity

2. Disclaimer…
This isn’t an attempt to persuade you to abandon pre-testing research!
It’s a discussion about how best to interpret and apply the conclusions
and recommendations that come out of pre-testing research.

3. Why are great brands so skeptical of pre-testing?
“We don’t ask “We do no market “We never pretested “Geoff believes
consumers what research. We figure anything we did at Nike, research is a blunt
they want. They out what we want. none of the ads. Dan instrument that
don’t know. Instead And I think we’re Wieden (the founder of bludgeons good
we apply our brain pretty good at Nike’s agency Wieden & ideas to death. He
power to what they having the right Kennedy) and I had an was determined that
need and will want discipline to think agreement that as long 42 Below would
and make sure we’re through whether a as our hearts beat, we never be subjected
there, ready.” lot of other people would never pretest a to pre-testing.”
are going to want it word of copy. It makes
- Akio Morita, Sony too. That’s what we you dull. It makes you - Justine Troy on
founder get paid to do.” predictable.” Geoff Ross,
founder of 42
- Steve Jobs, former - Scott Bedbury, Nike’s Below Vodka
Apple CEO former worldwide
advertising director

4. We humans aren’t great at picking successes…
“It’s a great invention, but who would want to use it?”
- US President Rutherford Hayes
Six years later, as the telephone was transforming America…
“The Americans have need of the telephone, but we do not. We have
plenty of messenger boys.”
- British Post Office’s Chief Engineer

5. We humans aren’t great at picking successes…
“Everyone acquainted with the subject will
recognize it (the light bulb) as a conspicuous
failure.”
- The President of Stevens Institute of Technology,
notable for producing several Nobel Prize winners,
1880

6. We humans aren’t great at picking successes…
“We don’t like their sound, and guitar
music is on the way out anyway.”
Decca Records’ leading A&R man
explaining his rejection of the
Beatles, 1962

7. We humans aren’t great at picking successes…
“The market researchers concluded that no other product had ever
performed so poorly in consumer testing: the look, taste and mouth-feel
were regarded as ‘disgusting’ and the idea that it ‘stimulates mind and
body’ didn’t persuade anyone that the taste was worth tolerating.”
- Philip Graves on Red Bull in his book Consumer.ology
In the two decades that followed, Red Bull sold over three billion cans of
its ‘disgusting’ drink, achieving sales of €2.6B.
Source: Consumer.ology by Philip Graves

8. We humans aren’t great at picking successes…
“Americans aren’t interested in Swedish vodka, with many people
unaware of where Sweden even is.”
- Conclusion of Manhattan’s Carillon Importers $80,000 Absolut
Vodka pre-testing research project
Absolut went on to sell over 70 million litres of vodka to the US
annually.
Source: Consumer.ology by Philip Graves

9. A history of pre-testing research
• Conceived in the 1950’s by American
psychologist Horace Schwerin
• He created a research product he called
‘persuasion testing’ and sold it to
advertisers as a way to measure the
potential sales impact of an
advertisement
• The method was analysed by university
researchers in 1965 and found to be
barely more reliable than flipping a coin.
Source: Excellence in advertising: the IPA guide to
best practice by Leslie Butterfield, p17

10. A history of pre-testing research
• In the 1990s, Beecham (now GSK), carried out
a long term global review of advertising testing
methods.
• They concluded “It ought to be emphasised
that no reliable pre-testing technique exists for
assessing the sales effectiveness of a specific
advertisement.”
• In 2004, researchers from the London
Business School noted that there was still “no
evidence in the public domain that pre-testing
is predictive.”
Source: Why Pre-Testing is Obselete by Tim Broadbent,
published in Admap Magazine, October 2004

11. A history of pre-testing research
Source: “Marketing in the Era of Accountability”
by Les Binet & Peter Field
• In 2007, the UK’s Institute of Practitioners in Advertising produced the largest ever study of historical marketing
effectiveness case studies (880 in total). They compared pre-tested campaigns with those that weren’t pre-tested.
• “Beware of pre-testing”, the study concluded. “If pre-testing really did lead to more effective campaigns, then one
would expect cases that reported favourable pre-testing outcomes to show bigger effects than those that did not. In
fact the reverse is true. Cases that reported favourable pre-testing results actually did significantly worse than those
that did not.”
• This shows that the judgement of marketers is in fact much more reliable than positive pre-testing outcomes

12. A history of pre-testing research
“I can never get a positive result. No matter how I cut the data, no
matter how I stack the odds in favour of pretesting by doing fine cuts
of the data, I can never get a result that says that pre-tested
campaigns are more effective than non-pretested campaigns.”
“If pre-testing really did work, we should at least get some positive
correlations, but we only ever get negative ones. After a while you
think, well, there’s an obvious conclusion to draw from all of that.”
- Peter Field, author of ‘Marketing in the Era of Accountability’
Source: Interview with the author

13. Equally, there are all sorts of pre-testing successes
Among others, Cadbury Gorilla and Old Spice flew
through pre-testing, and went on to become highly
effective campaigns.
It isn’t that pre-testing always gets it wrong.
It’s just very difficult to predict whether the pre-testing
conclusions are right or wrong.
So why is pre-testing so fickle?

14. We are biased toward the familiar
American social psychologist Robert Zajonc studied what he called ‘the
exposure effect’ in the 1970s. His experiments showed that simply
exposing subjects to a familiar stimulus led them to rate it more
positively than other, similar stimuli that had not been previously
presented.
In one experiment, people were shown a random sample of squiggle
drawings. Some time later, they were shown the same sample, but this
time the squiggles were placed randomly among a selection of other,
similar squiggles. The subjects were asked whether they could
remember which of the squiggles were the ones they were previously
shown. As you’d expect, they had a hard time with the exercise and
rarely chose correctly. Then they were asked to show the interviewer
which squiggles they preferred. They found this test considerably
easier, and unbeknownst to them, chose the squiggles that they’d seen
the first time around.
Zajonc’s work concluded that people tend to develop a preference for
things merely because they’re familiar with them.
Source: Affective Discrimination of Stimuli
That Cannot Be Recognized”, Kunst-Wilson &
Zajonc, published in Science, Vol. 207

15. We tend to be wrong about what we think we want
Google asked customers how many results they
wanted the search engine to throw back on the first
page.
“Since conventional wisdom says more is always better,
people naturally said ‘more’. When Google tripled the
number of results, however, it found that traffic actually
declined. Not only did the results take a fraction of a
second longer to load, but having more options led
people to click on links that were less relevant. The
respondents in Google’s research didn’t intentionally
lead researchers down the wrong path; they just didn’t
understand the real-world implications of their choices.”
- Steve McKee, BusinessWeek, 2010.

16. Often things we dislike ‘grow on us’
There’s an example of this effect that most of us are familiar with. Upon initially listening to a new album, we prefer
certain songs to others. On subsequent listens this preference usually changes, and our long term favourite songs
tend not to be the ones we liked at first.
"It is easy to slip into the comfortable belief that 'I like it' comments after the first exposure of a new execution are a
must. In researching Levi's executions over the years, it has become abundantly clear that such findings should be
treated with extreme caution.”
- Kirsty Fuller, Managing Director of RDS International Research, in 1995.
Source: Walking the creative tightrope: the research challenge”, Kirsty Fuller, published in Admap Magazine, March 1995

17. Often things we dislike ‘grow on us’
eg Levi’s ‘Swimmer’
"In pre-launch qualitative research, response from the consumer on first exposure
to Swimmer was one of stunned silence. The hero's status was initially seen to be
seriously undermined - he did little to earn his colours. Moreover the slow music
did not have the immediate appeal of previous executions. Perhaps the most fitting
description of response was disappointment. Swimmer broke the mould of the
campaign to date, and consumers claimed not to like it.”
“At this stage the weight of negative reactions was strong. Then two months after
airing, research uncovered a marked shift in response. Swimmer had become a
talking point: new, different, challenging. A further four months later and Swimmer
was being widely described as one of the best ever Levi's ads, destined to live
among the greats, such as the universally acclaimed Launderette.”
“Research must therefore seek to evaluate the potential of an execution, not its
immediate impact on one viewing. Challenging advertising is not necessarily either
immediately liked or fully understood. It may however, be rich and long-lasting.”

18. So…
• A long history of pre-testing research being studied and proven unreliable
• Pre-testing often gets it right, but it’s very difficult to predict when that will be the case
• Pre-testing is hampered by a few inconvenient realities…
• We’re biased toward the familiar, not the effective
• We often think (and will report) we want things that we actually don’t
• Pre-testing only offers a ‘first impression’ whereas new ideas tend to ‘grow on us’
• The numbers show that marketers’ judgment is significantly more reliable than positive pre-testing outcomes

19. The case for moderation
Alcohol has positive and negative effects.
When we use it moderately, it’s great.
When we use it immoderately, we get into trouble.
Pre-testing is the same.
It’s useful when used as part of a wider decision and
development process
But when used ‘immoderately’ – as a decision maker –
it’s dangerous.
Let’s be moderate in how we use the outcomes from
our pre-testing research.