Measure Right The First Time

MEASURE RIGHT
THE FIRST TIME
Scott Yacko & Jonathan Kressaty
BOLO 2012

vuurr.com

1

WHAT WE’LL COVER
_ What you need
_ How to track
_ How to design experiments
_ What to experiment on
_ Putting it all together

2

INTERNET MARKETING IS
THE INTERSECTION OF
COPYWRITING AND
MATHEMATICS.

3

YOUR HIRING DECISIONS
SHOULD REFLECT THIS.

4

TOOLS YOU MUST HAVE
_ Google Analytics (or SiteCatalyst, or similar)
_ Optimizely (or similar, but you need killer statistics)
_ Call tracking => analytics platform of choice
_ Link builder appropriate for your analytics platform of choice
_ AdWords/AdCenter/Facebook properly conﬁgured (if advertising)

6

DETERMINE YOUR MODEL

CPA ROI MARGIN

Minimize the Maximize Minimize ad
cost of each revenue for spend as a
customer every dollar percent of
acquisition spent revenue

7

DETERMINE YOUR MODEL

CPA ROI MARGIN

Ad Spend ($) Revenue - Spend Revenue - Spend
Conversion (qty) Spend Revenue

answer is in dollars ($) per conversion all in dollars ($), answer is % all in dollars ($), answer is %

8

DETERMINE YOUR MODEL -
FINDING ROI & MARGIN

MARGIN

ROI

9

FINDING CPA (MANUALLY)
Completed
Advertising => AdWords => Campaigns => Site Usage Conversions

Advertising => AdWords => Campaigns => Clicks
Total Cost

10

FINDING CPA (AUTOMATICALLY)
_ Use the Google Analytics API to query for total number of
conversion by source & medium, and then properly divide
them into the total marketing expense for each source &
medium.
_ Use a third party tool

11

INSTALLING ANALYTICS
_ Follow your vendor’s instructions
_ Don’t just dump the tracking code in and “go”
_ Setup proper funnels, events, goals, and ﬁlters
_ Know the difference between accounts, properties,
and proﬁles
_ Attach GWT and AdWords (if applicable)
_ Track ALL OF THE THINGS

13

GOALS & ECOMMERCE
_ Assign relevant goal values - video view != purchase
_ Track eCommerce as accurately as possible
_ Explicit funnels are only useful if the supporting steps
are relevant and provide information - use sparingly

14

TRACKING CAMPAIGNS

_ Every click is valuable
Name/
Content _ Every trafﬁc source matters
_ Proper campaign tracking is
Campaign
crucial to adequately gauging
Source Medium
trafﬁc value

15

PROPER URL TAGGING

16

TRACKING RICH MEDIA:
VIDEOS

_ Instantiate video with YouTube/Vendor JavaScript API
_ Track state changes, namely Plays and Completes
_ On these changes, push GA events

17

SOCIAL BUTTONS
_gaq.push(['_trackSocial', network, socialAction, opt_target,
opt_pagePath]);

Instantiate by subscribing to edge.create and
message.send

Google Analytics handles for you

Instantiate by subscribing to the ‘tweet’ intent event

18

FORM ABANDONMENT
_ Track full and empty exits as events
_ Determine ﬁelds causing abandonment

19

TRACKING OFFLINE:
PHONE CALLS
_ Build GA gif request and HTTP GET on phone call
_ Push calls as pageviews ( /call/+14805551234/abcdefg )
_ Use Twilio & Twimlbin or another 3rd party service
_ Consider filtering out those pageviews from main profile
_ Setup goals in Google Analytics to fire on calls
_ Give each ad, landing page, social source, different number

20

TRACKING PAID CLICKS
_ Properly conﬁgure AdWords to push to Analytics
_ Verify all cost data is applied to Analytics account
_ Put MSN AdCenter {AdId} & {Keyword} in destination URL,
then use AdCenter API to pull the cost and associate with
relevant clicks from Analytics API

21

EXPERIMENT DESIGN

22

INTERNET MARKETING IS
ABOUT EXPERIMENTATION
y = f( x1, x2, x3, . . . . xn )
y= CTR x1 = Page or Ad Copy

CR x2 = Page Layout

Goals x3 = Button Color & Size

xn = Other Factors...

23

DESIGNING EXPERIMENTS -
ONE FACTOR AT A TIME
Let’s run an experiment where we test two different button colors.

Run Button Color ( x1 ) CTR ( y )
1 Green 5.2%

2 Orange 7.7%

Factors are tested in series, one at a time. This is extremely time
intensive, requiring many tests for very gradual improvement.

24

FULL FACTORIAL DESIGN
Let’s run an experiment where we test two different button colors,
button positions, and button labels simultaneously.

3 FACTORS WITH 2 VARIATIONS EACH

Number of Unique Tests =23=8

This tests multiple changes in parallel to ﬁnd optimal combination of
factors (x variables) to optimize results (y variable).

25

FULL FACTORIAL DESIGN
Run Color (x1) Position (x2) Text (x3) CTR (y)
1 Green Left Try Now 5.2%
2 Orange Left Try Now 7.7%
3 Green Right Try Now 9.3%
4 Orange Right Try Now 12.7%
5 Green Left Buy Today 21.2%
6 Orange Left Buy Today 19.3%
7 Green Right Buy Today 14.9%
8 Orange Right Buy Today 17.7%
26

OFAT VS. FULL FACTORIAL
ONE FACTOR AT A TIME FULL FACTORIAL
Iterative More Complex to Setup
Simple to Setup VS. Requires more visitor data
Easy to Measure Results to test all combinations

SLOW & GRADUAL QUICK & DRASTIC
IMPROVEMENT IMPROVEMENT

27

STATISTICAL SIGNIFICANCE
Preliminary results - is orange better?

Button Color ( x1 ) Visitors Clicks CTR ( y )
Green 38 2 5.2%

Orange 39 3 7.7%

Too small of a sample size leads to incorrect conclusions.

28

STATISTICAL SIGNIFICANCE
It turns out our original conclusion of CTR was incorrect.

Button Color ( x1 ) Visitors Clicks CTR ( y )
Green 8,238 486 5.9%

Orange 7,893 734 9.3%

Too large of a sample size wastes time, effort, & money.

29

SO WHAT QUANTITY OF
VISITORS IS ENOUGH?

30

DETERMINING MINIMUM
SAMPLE SIZE
x = Sample mean
µ = Population mean E=x-µ
Z = Number of standard
deviations above mean
α = Conﬁdence distance
σ
from 100%
σ = Standard deviation E = Z α/2 √n
n = Minimum sample size

31

DETERMINING MINIMUM
SAMPLE SIZE

- α/2 Z α/2

32

DETERMINING MINIMUM
SAMPLE SIZE (CONTINUED)
σ Standard

E = Z α/2 √n Deviation

Minimum
Sample Size

Z σ
n= [ α/2

E ]
33

A/B TESTING SOFTWARE

34

EVERY SINGLE CLICK HAS A
COST ASSOCIATED WITH IT

36

KNOW YOUR LEVERS
_ Product Pricing
_ Organic Rankings
_ Ad bids, placement, content
_ Email subject lines & content
_ Social content
_ Site Content

37

FUNNEL ANALYSIS &
OPTIMIZATION
IMPRESSIONS Trafﬁc Sources drive impressions & clicks

CTR _ SEM, SEO, Social, Email, & More

Optimizing those sources get more clicks
CLICKS _ Improve Quality Score
(Site Visits) _ Split Test Ad Copy
_ Adjust Keyword Bids
_ Increase Organic Rankings
_ Increase Social Reach
CR
Optimized landing pages & site content increase conversions
_ Split Test Landing Page Copy
_ Improve Call to Action
_ Experiment with Deep Linking
CONVERSIONS _ Look at Bounce Rates
_ Look at Cart Abandonment
(Sales/Leads)

38

FUNNEL ANALYSIS &
OPTIMIZATION
ORGANIC OFFLINE
EMAIL SEARCH MARKETING AFFILIATES
PPC SOCIAL PHONE CALLS
ALL OF THESE SOURCES HAVE
LEVERS THAT MANIPULATE THEIR
QUALITY & LIKELIHOOD
TO BECOME
BUYERS

39

EMAIL OPTIMIZATION LEVERS
EMAILS SENT
LEVER 1: List Size Increase List Size
_ Completely _ Optimize CR on
dependent on subscription page
conversion rate of _ Increase sales to lure
OPENED subscription form repeat buyers
EMAILS LEVER 2: Open Rate Test Subject Line
LEVER 3: CTR Test Email Copy

LINKS
CLICKED
40

EMAILS SENT
LEVER 2: Open Rate Test Subject Line
Quantity Opened _ Use an email application
OPEN =
OPENED with A/B testing
RATE Quantity Sent _ Call to action is
EMAILS important

LINKS
CLICKED
41

EMAILS SENT
LEVER 2: Open Rate Test Subject Line
OPENED Clicks from Email _ Treat as a landing page
EMAILS CTR = Quantity Opened _ Test layout, buttons,
content, etc.

LINKS
CLICKED
42

PPC OPTIMIZATION LEVERS
AD IMPRESSIONS LEVER 1: Budget Increase Budget
_ Directly drive _ Optimize for maximum
conversions factoring in conversions at current CPC
conversion rate & CPC
LEVER 2: Bids Optimize for KPI
AD CLICKS
LEVER 3: Quality Score Maximize QS
LEVER 4: CTR Test Ad Copy

43

Ranking Score of Position Below
Actual
Cost = Quality Score of Your Ad
+ $0.01
AD CLICKS

44

_ Function of ad content, _ Relevant ad
AD CLICKS landing page content, _ Relevant landing page
historical CTR, & more _ Display URL

45


AD CLICKS LEVER 4: CTR Test Ad Copy
Clicks from Ad _ Continuously A/B test ad
CTR = Ad Impressions _ Keep highest CTR ad
_ Delete lower CTR ads
_ Repeat until returns
diminish

46

SOCIAL CONTENT
OPTIMIZATION LEVERS
SOCIAL
IMPRESSIONS
LEVER 1: Network Add Networks
_ Be there in the ﬁrst place _ Pinterest
_ Reddit
_ Others
SOCIAL CONTENT
CLICKS LEVER 2: Content Quantity/Quality
LEVER 3: Interactions Followers/Frequency

47

SOCIAL CONTENT
OPTIMIZATION LEVERS
SOCIAL
IMPRESSIONS
LEVER 2: Content Quantity/Quality
_ Links to deep content _ Frequency
_ Polite, relevant posts _ Better content drives
SOCIAL CONTENT more shares/retweets
CLICKS

48

SOCIAL CONTENT
OPTIMIZATION LEVERS
SOCIAL
IMPRESSIONS
LEVER 2: Content Quantity/Quality
SOCIAL CONTENT _ More interactions drives _ Increase quantity and
CLICKS more clicks value of followers
_ Don’t “buy” followers

49

WEBSITE LAYOUT & CONTENT

VISITS LEVER 1: Layout Forms, Phone #s
LEVER 2: Content Quantity, Quality
LEVER 3: Click Areas Size, Color, Position
CONVERSIONS LEVER X: etc.....

50

THE WEB IS A SYSTEM
_ Your marketing strategy is a system of equations
_ Some levers are independent of others
_ Some levers manipulate others
_ Some levers have a larger impact on KPI than others

Ad Bid SEO Email Social
Spend Subject Effort
52

THE WEB IS A SYSTEM
_ Think about moving several levers at once
_ Always consider what a single and multiple-lever move will do to
the system, and accordingly, your end result
_ Design a good experiment, continue to measure, and repeat

y = f( x1, x2, x3, . . . . xn )
53

EXAMPLE SCENARIO
_ Current Budget: $7,500 / month ( $250 / day )
_ Conversions: 48 / month ( 1.6 / day )
_ Average CPA = $158.17
_ Average CPC = $2.29
_ Landing Page CR = 1.45%

New Target: 10 Conversions / Day AND Lower CPA

54

SCENARIO ONE:
ADJUSTING BUDGET
_ Average CPA at $158.17 means 10 conversions per day is $1581.70
_ $47,000 per month
_ Easy, but too expensive

55

SCENARIO TWO:
ADJUSTING CPC
_ Budget stays at $250 / day, conversion rate steady at 1.45%
_ Need 689 clicks to get 10 conversions
_ Average CPC must be $0.36
_ Improbable to bring $2.29 down to $0.36 in this space while
maintaining volume

56

SCENARIO THREE:
ADJUST CONVERSION RATE
_ Budget stays at $250 / day, CPC at $2.29
_ Can purchase 110 clicks for $250
_ To get 10 conversions, landing page needs to convert at 9.1%
_ Improbable to quickly jump from 1.45% to 9.1%

57

FINAL SCENARIO:
MOVE ALL THREE LEVERS
_ Double the budget to $15,000
_ Reduce average CPC by 50% ( $2.29 to $1.15 ) via QS & Keywords
_ Improve landing page CR from 1.45% to 2.3% via A/B testing
_ ( $15k / $1.15 per click ) at 2.3% drives 300 conversions monthly at
a CPA of $50
_ Used 3 levers simultaneously to improve conversions
6x and reduce average CPA by 68%

58

CREATE A CULTURE OF
CONTINUOUS IMPROVEMENT
_ Deﬁne success
_ Measure current performance and compare to targets
_ Determine biggest levers and how they contribute to KPIs
_ Design an experiment
_ Test repeatedly
_ Finalize improvements based on adequate data
_ Repeat
60

QUESTIONS?
Scott Yacko - scott@vuurr.com - @scottmyacko
Jonathan Kressaty - jonathan@vuurr.com - @kressaty

Code & Presentation at
vuurr.com/bolo
61

Measure Right The First Time

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (9)

Similaire à Measure Right The First Time

Similaire à Measure Right The First Time (20)

Dernier

Dernier (20)

Measure Right The First Time