FOX uses Optimizely's full platform solutions to support experimentation across its product and marketing teams. FOX relies on experimentation to drive key business metrics for its brands like FOX Now, FOX Sports, and FOX News. The Director of Digital Strategy and Optimization at FOX's Growth Marketing team oversees experimentation efforts to improve subscription signups, video and content engagement, and pay-per-view purchases. FOX Product teams also leverage Optimizely to test product updates, inform decisions, and validate hypotheses. The Director of Product Management leads an Optimization Guild to help FOX product teams design, run and analyze experiments.
How FOX Tests Everything from Mobile, Web, to Living Room Devices
1. 1
How FOX Tests Everything
from Mobile, Web, to Living
Room Devices
Sara Miller & Paul Tongyoo
January 2020
2. Housekeeping
● Customize the widgets on your page to
your preference
● This webinar is recorded and you will
receive the link with the slides in the next
few days
● We will have time for questions! Please
submit them in the Q&A box on the left
side of the screen
3. 3
Paul Tongyoo
Director, Product Management
Sara Miller
Director, Optimization and Digital Strategy
Introductions
Mei Luo
Strategic Customer Success Manager
6. Revenue, share of wallet, funnel conversion, risk
mitigation, ops efficiency
What
We Do Next gen “Test and Learn” platform for enterprise-
wide digital experience optimization
Over 1B impressions daily
Replaces Digital Guesswork
With Evidence-based Optimization
Digital Experimentation
SaaS Platform
Apply the scientific method to “at-scale”
business decision making.
20X increase in Yield
8. The digital-centric
customer journey: end-to-
end, multi-device
engagement
Marketing Product & Engineering Support
Views ad
at Checkout
Signs up for
7 day trial
Tracks
packages
Manages
account
Converts to
subscriber
Downloads
App
9. Frontend UI Backend Business
Logic & Data
The anatomy of an experience
i.e. navigation,
search location
& visual treatment
Copy
Images &
Colors
Layout Search algorithms
Personalize content
based on previous
behavior
Recommendations
Make your headlines
more personal
10. “Our success is a function of
how many experiments we do
per year, per month, per week,
per day.”
Jeff Bezos
“Our aim is to create the best
product for our customers, and
we do that through constant
innovation and testing.”
Gillan Tans, CEO
“Our company culture
encourages experimentation
and free flow of ideas.”
Larry Page
“We use experimentation and
testing to inform as much of the
business as we possibly can” -
Gregory Peters, CPO
Today’s Digital Leaders Win By
Using Experimentation At-Scale
11. Spent with Digital Media
19 hr
week
2018
Digital-Centric Customer
Spent with Digital Media
41
$396m
Mobile commerce Mobile commerce
$1 8t.
2008
Digital-Convenient Customer
hr
week
Your customer has fundamentally
changed how they engage
13. We’re Proud to Work With Great Global Enterprises
24 of the Fortune 100 have chosen Optimizely to drive their digital experience
14. FOX Leverages Optimizely to
Foster a Data-Driven Culture
FOX relies on Optimizely’s full platform solutions
to support experimentation in product and
marketing teams among brands like:
● FOX Now
● FOX Sports
● FOX News
To drive key business metrics like:
● Video & Content Engagement
● Subscription Signups
● Pay-Per-View Purchases
16. Topics
● About Me
● My Experimentation Philosophy
● Plans to Grow and Scale
● Advice to Other Marketers
● How Marketing and Product
Experimentation Works Together
17. ABOUT ME
17
Director of Digital
Strategy and
Optimization
Experience in Marketing
and Product
8+ yrs CRO and
Experimentation
Experience
Avid sports fan
Experimented with a
variety of tools
22. My Experimentation Philosophy
22
The real world isn’t a perfectly controlled environment,
so you may have give up certain things in order to move
at the pace the business needs.
23. Pre-Experiment analysis is more
important than post analysis*
*Post experiment analysis communication is the most important of all
24. My Experimentation Philosophy
24
If you do the work in pre-experiment analysis, your post
experiment analysis is easy and you can avoid
inconclusive results and wasted resources.
25. Being wrong is a good thing*
*The more wrong you are in the beginning, the more right you become later
28. FOX Growth Marketing continues to grow...
28
Continue to
optimize key
User Journeys
More Personalization
to support array of
FOX digital users
Capitalize on Major
Events to drive key
business objectives
Expand Optimization
onto Mobile and
Connected Devices
30. How other marketers can grow and scale
30
Start with Low-
Hanging Fruit
Focus on
Business
Objectives for
Prioritization
Insist on dedicated
resources for
Optimization
Cast a wide net
for brainstorming
and ideation
31. How Marketing and Product Experimentation Work Together
31
Ideation (Marketing)
Exploratory experiments to
validate research/analysis
based hypotheses
Example
Product/Feature
Development
Cycle
Definition/Planning
(Product)
Exploratory experiments for
direction and to inform
prioritization
Development (Product)
QoS experiments to prevent unintended
consequences of feature design
Release (Product)
Validation experiments before
full launch to validate expected
impact
Monitor/Optimization
(Marketing)
Optimization experiments
to personalize experiences,
optimize user journeys,
increase adoption, etc.
33. 3 3
WHO AM I?
H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
34. 3 4
Who am I?
• A Director of Product Management focusing on Optimization
• Leads Optimization Guild to help anyone at FOX use experimentation
• Software Engineer before Product Manager
• About 7 years at FOX managing Web, Mobile, Analytics, Optimization products
H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Paul Tongyoo
35. 3 5H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
PRODUCT TEAM OVERVIEW
36. 3 6
How to help
all these PMs
learn from
testing?
Web
Mobile
OTT (Native)
OTT (Web-based)
Sports Web
Sports Mobile
H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Pilot Product Team: Los Angeles
Commerce & Customer Care
Profile & Personalization
Video
SDK
CMS
and more…
37. 3 7H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
PROCESS DISCOVERY
39. EXPERIMENTATION PROCESS
PRODUCT
MANAGER
EV
Product Managers couldn’t prioritize time for:
H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Strategy #1: Enablement
What I Learned
• Design & planning
• Test configuration & management
• Info sharing across organization
• Impression governance
40. EXPERIMENTATION PROCESS
PRODUCT
MANAGER
OPTIMIZATION
GUILD
H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Strategy #2: Service
PROJECT
TIMEFRAME
WRITE BRIEF
WRITE DEV
GUIDE
DEV & QA
CONFIG/RUN
EXPERIMENT
KEEP ORG INFORMED
MONITOR IMPRESSIONS
DEFINE 4
W’s*
DERIVE
LEARNING
S
*What, Why, Who, and What Metrics
47. 4 7
EXPERIMENTATION PROCESS
PRODUCT
MANAGER
OPTIMIZATION
GUILD
PROJECT
DURATION
WRITE BRIEF WRITE DEV GUIDE
DEV & QA
CONFIG/RUN
EXPERIMENT
KEEP ORG INFORMED
MONITOR IMPRESSIONS
DEFINE 4
W’s
DERIVE
LEARNINGS
H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Service Strategy Observations
• Increased engagement
• Increased test-informed decisions
• Increased experimentation expertise
• Foundation for future enablement
48. 4 8H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
2020 GOALS
49. 4 9H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
2020 Goals
• "Experiment Running on Every Device”
• Enable Engineering Team Experimentation
• Build Pre-Experiment Analysis Tools
• Expand beyond Los Angeles
50. 5 0H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
EXAMPLE EXPERIMENTS
51. 5 1
New DesignControl What: Test new Series Page design
Why: Current UI “not clear”, Low CVRs
Who: All Series Page Traffic
What Metrics: CVRs for Notification Opt-in,
Favoriting, Starting Video
How: Implemented one Optimizely “Live
Variable” to toggle visibility of new design
What We Learned: New Design resulted in stats-
sig CVR increases across all Metrics:
• Global Notifications: +28%
• Series Notifications: +64%
• Favoriting Series: +15%
• Starting Video: +7%
H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Test to Validate Product Updates
52. 5 2H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Test to Inform Decisions
53. 5 3H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Test to Inform Decisions
54. 5 4H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Add
Apple Sign In
Test to Inform Decisions
55. 5 5H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Add
Apple Sign In
Remove
Facebook
Test to Inform Decisions
56. 5 6H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
What effect does removing Facebook
have on sign up rate?
Test to Inform Decisions
57. 5 7H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
KPI
Effect of Removing
Facebook
Decision
Sign Up Rate Stays Flat or Increases Remove Facebook
Sign Up Rate Decreases Add Apple Sign In
Test to Inform Decisions
58. 5 8H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Control No Facebook
Sign Up Form
Test to Inform Decisions
59. 5 9H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Control
No
Facebook
Sign Up
Form
Sign In
Form
How to
implemen
t this?
“New”
Users
Test to Inform Decisions
“Existing”
Users
Experiment
60. 6 0H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Sign Up Form
2. Call
activate(…)
method
Sign In Form
3. Call
getVariation(…)
method
1. Integrate
Optimizely
Full Stack
SDK
Test to Inform Decisions
61. 6 1H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
ADVICE TO PRODUCT MANAGERS
62. 6 2H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Advice to Product Managers
• Changing the UI is not enough. Validate your test’s data and statistical power as well.
• Evaluate the credibility, accessibility, and usability of your data. Can you improve?
• New to statistics? Pick up a book! Will motivate, rationalize, and refine your testing efforts.
63. 6 3H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
THANKS!