SlideShare une entreprise Scribd logo

Why everything is an A/B Test at Pinterest

•
•
Krishna Gade
Krishna Gade

A/B Experimentation at Pinterest

Données & analyses
1  sur  48
Télécharger pour lire hors ligne
Krishna Gade Why everything is an A/B test at Pinterest
Confidentia l 2
What is Pinterest? A visual bookmarking tool Discover an inspiring idea Save it to a board Go do it
Pinterest is a data product
Data Flow App events Kafka Secor Skyline Pinball Redshift Pinalytics Features Qubole (Hadoop) Singer
Confidentia l Metrics 7
Confidentia l User Lifecycle New Users Active Users Dormant Users Acquisition Activation Churn Churn Resurrection
Confidentia l Terminology ● MAU – Monthly Active User ● WAU – Weekly Active User ● DAU – Daily Active User ● WAR – Weekly Active Repinner ● WAC – Weekly Active Clickthrougher ● WARC – Weekly Active Repinner or Clickthrougher ● 1d7 – User who comes back at-least once in 7 days after signup ● 1rc7 - User who repins/clicks at-least once in 7 days after signup
Confidentia l MAUs ● Current progress to quarterly goal ● MAU forecast ● MAUs by app/gender/country ● MAU Accounting
Confidentia l MAU Accounting
Confidentia l Acquisition Metrics ● Total Signups ● Signups by App ● Signups by Referrer ● Invites Sent ● Unique Invite Senders ● Invite Signups
Confidentia l Activation Metrics Activation Rate = 1rc7/signups ● Activation Rate by App/Gender/Referrer ● Overall signups to 1rc7 ● Signup engagement funnel ● Signups to WAU 35 days after signup
Confidentia l Engagement Metrics ● *AU ratios ● # of Emails sent ● # of Emails opened ● # of Emails clicked ● # of Notifications sent ● # of Notifications clicked
Confidentia l Resurrection Metrics ● # of Users who were dormant for 28+ days but came back, split by platform, referrer etc
Confidentia l Takeaways ● Emphasize on activation (process of converting a new user to MAU) ● Activation critical to long-term growth ● Deeply understand how different segments of users interact
Confidentia l Automated Metrics 17
Confidentia l
Confidentia l Chart creation
Confidentia l Reuse existing query result
Confidentia l Metrics creation
Confidentia l A/B Experimentation 22
Confidentia l Categories of Experiments ● A/B testing new features ● Copy testing ● SEO changes ● Infrastructure updates
Confidentia l A/B Testing Framework
Confidentia l
Confidentia l
Confidentia l
Confidentia l A/B Testing Dashboard
Confidentia l
Confidentia l p-value (via chi-squared test) Statistical tests for validation
Confidentia l Impact of Badging
Confidentia l Impact of Badging ● 7% lift in DAU ● 2.5% lift over long term ● Effective at increasing engagement levels
Confidentia l Copy Testing Definition: Testing different variations of text we display to the user. ● A/B Testing copy was a tedious process ● Even when we did this we were optimizing only for English
Confidentia l New Approach ● Ability to run a copy test w/o changing a line of code. ● Support for translating text dynamically ● Use Multi-armed bandits to converge to showing best performing text
Confidentia l Multi Armed Bandits
Confidentia l SEO 36 50+ Billion Pins categorized by people into more than 1 Billion Boards
Confidentia l 37 What makes SEO so challenging? What is the Impact of the new SEO feature? Will this new change affect SEO? Uhhh... Dunno... Uhhh... Ask Google.
Confidentia l KEEP CALM and RUN AN SEO EXPERIMENT
Confidentia l What is an SEO experiment? 39 Regular AB Experiment SEO Experiment Metrics User behavior Search engine referred traffic Unique id for hashing userid / sessionid page url
Confidentia l What is an SEO experiment? Launch ● Measure the gap ● Before vs After launch
Confidentia l Dashboard 41 DIFF = enabled - control Traffic Gain
Confidentia l 42 Decision Making 1 2 3 4 SHIP! SHIP! SHUT DOWN! VALIDATE
Confidentia l 43 Experiment Driven SEO What is the Impact of the new SEO feature? We got XX% of traffic growth from it! Can we use more resources to make bigger impact? No, the experiment result is flat, so let’s ship it. SEO experiment SEO experiment Is my UI change affecting SEO negatively?
Confidentia l Why you should be A/B testing even your infrastructure
Confidentia l Infrastructure Changes ● 2013: A new web framework ● 2014: Pyapns ● 2015: User service
Confidentia l Experiment results for replacing PyAPNs
Confidentia l Conclusions ● Figure out what success means ● Define your metrics ● Automate metric computation ● A/B test everything!
Confidentia l Thank You!

Recommandé

A/B testing at Spotify
A/B testing at SpotifyA/B testing at Spotify
A/B testing at SpotifyAli Sarrafi
 
A/B Testing with Yammer's Product Manager
A/B Testing with Yammer's Product ManagerA/B Testing with Yammer's Product Manager
A/B Testing with Yammer's Product ManagerProduct School
 
A/B Testing Pitfalls and Lessons Learned at Spotify
A/B Testing Pitfalls and Lessons Learned at SpotifyA/B Testing Pitfalls and Lessons Learned at Spotify
A/B Testing Pitfalls and Lessons Learned at SpotifyDanielle Jabin
 
SAMPLE SIZE – The indispensable A/B test calculation that you’re not making
SAMPLE SIZE – The indispensable A/B test calculation that you’re not makingSAMPLE SIZE – The indispensable A/B test calculation that you’re not making
SAMPLE SIZE – The indispensable A/B test calculation that you’re not makingZack Notes
 
A/B Testing Framework Design
A/B Testing Framework DesignA/B Testing Framework Design
A/B Testing Framework DesignPatrick McKenzie
 
Basics of AB testing in online products
Basics of AB testing in online productsBasics of AB testing in online products
Basics of AB testing in online productsAshish Dua
 
Ab testing 101
Ab testing 101Ab testing 101
Ab testing 101Ashish Dua
 
Talks@Coursera - A/B Testing @ Internet Scale
Talks@Coursera - A/B Testing @ Internet ScaleTalks@Coursera - A/B Testing @ Internet Scale
Talks@Coursera - A/B Testing @ Internet Scalecourseratalks
 

Recommandé

A/B testing at Spotify
A/B testing at SpotifyA/B testing at Spotify
A/B testing at SpotifyAli Sarrafi
 
A/B Testing with Yammer's Product Manager
A/B Testing with Yammer's Product ManagerA/B Testing with Yammer's Product Manager
A/B Testing with Yammer's Product ManagerProduct School
 
A/B Testing Pitfalls and Lessons Learned at Spotify
A/B Testing Pitfalls and Lessons Learned at SpotifyA/B Testing Pitfalls and Lessons Learned at Spotify
A/B Testing Pitfalls and Lessons Learned at SpotifyDanielle Jabin
 
SAMPLE SIZE – The indispensable A/B test calculation that you’re not making
SAMPLE SIZE – The indispensable A/B test calculation that you’re not makingSAMPLE SIZE – The indispensable A/B test calculation that you’re not making
SAMPLE SIZE – The indispensable A/B test calculation that you’re not makingZack Notes
 
A/B Testing Framework Design
A/B Testing Framework DesignA/B Testing Framework Design
A/B Testing Framework DesignPatrick McKenzie
 
Basics of AB testing in online products
Basics of AB testing in online productsBasics of AB testing in online products
Basics of AB testing in online productsAshish Dua
 
Ab testing 101
Ab testing 101Ab testing 101
Ab testing 101Ashish Dua
 
Talks@Coursera - A/B Testing @ Internet Scale
Talks@Coursera - A/B Testing @ Internet ScaleTalks@Coursera - A/B Testing @ Internet Scale
Talks@Coursera - A/B Testing @ Internet Scalecourseratalks
 
4 Steps Toward Scientific A/B Testing
4 Steps Toward Scientific A/B Testing4 Steps Toward Scientific A/B Testing
4 Steps Toward Scientific A/B TestingJanessa Lantz
 
SXSW 2016 - Everything you think about A/B testing is wrong
SXSW 2016 - Everything you think about A/B testing is wrongSXSW 2016 - Everything you think about A/B testing is wrong
SXSW 2016 - Everything you think about A/B testing is wrongDan Chuparkoff
 
Practical Introduction to A/B Testing
Practical Introduction to A/B TestingPractical Introduction to A/B Testing
Practical Introduction to A/B TestingAlex Alwan
 
A/B Testing Best Practices - Do's and Don'ts
A/B Testing Best Practices - Do's and Don'tsA/B Testing Best Practices - Do's and Don'ts
A/B Testing Best Practices - Do's and Don'tsRamkumar Ravichandran
 
A/B Testing at Pinterest: Building a Culture of Experimentation
A/B Testing at Pinterest: Building a Culture of Experimentation A/B Testing at Pinterest: Building a Culture of Experimentation
A/B Testing at Pinterest: Building a Culture of Experimentation WrangleConf
 
A/B testing in Firebase. Intermediate and advanced approach
A/B testing in Firebase. Intermediate and advanced approachA/B testing in Firebase. Intermediate and advanced approach
A/B testing in Firebase. Intermediate and advanced approachGameCamp
 
10 Guidelines for A/B Testing
10 Guidelines for A/B Testing10 Guidelines for A/B Testing
10 Guidelines for A/B TestingEmily Robinson
 
The Scientific Method of Experimentation by Google PM
The Scientific Method of Experimentation by Google PMThe Scientific Method of Experimentation by Google PM
The Scientific Method of Experimentation by Google PMProduct School
 
Music Personalization At Spotify
Music Personalization At SpotifyMusic Personalization At Spotify
Music Personalization At SpotifyVidhya Murali
 
Intro to A/B Testing by Ever's Senior Product Manager
Intro to A/B Testing by Ever's Senior Product ManagerIntro to A/B Testing by Ever's Senior Product Manager
Intro to A/B Testing by Ever's Senior Product ManagerProduct School
 
AB Testing at Expedia
AB Testing at ExpediaAB Testing at Expedia
AB Testing at ExpediaPaul Lucas
 
Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.js
Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.jsNetflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.js
Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.jsChris Saint-Amant
 
Recommending and searching @ Spotify
Recommending and searching @ SpotifyRecommending and searching @ Spotify
Recommending and searching @ SpotifyMounia Lalmas-Roelleke
 
Machine Learning and Big Data for Music Discovery at Spotify
Machine Learning and Big Data for Music Discovery at SpotifyMachine Learning and Big Data for Music Discovery at Spotify
Machine Learning and Big Data for Music Discovery at SpotifyChing-Wei Chen
 
Building an A/B Testing Analytics System with R and Shiny
Building an A/B Testing Analytics System with R and ShinyBuilding an A/B Testing Analytics System with R and Shiny
Building an A/B Testing Analytics System with R and ShinyEmily Robinson
 
Netflix talk at ML Platform meetup Sep 2019
Netflix talk at ML Platform meetup Sep 2019Netflix talk at ML Platform meetup Sep 2019
Netflix talk at ML Platform meetup Sep 2019Faisal Siddiqi
 
The evolution of the big data platform @ Netflix (OSCON 2015)
The evolution of the big data platform @ Netflix (OSCON 2015)The evolution of the big data platform @ Netflix (OSCON 2015)
The evolution of the big data platform @ Netflix (OSCON 2015)Eva Tse
 
Building Data Pipelines for Music Recommendations at Spotify
Building Data Pipelines for Music Recommendations at SpotifyBuilding Data Pipelines for Music Recommendations at Spotify
Building Data Pipelines for Music Recommendations at SpotifyVidhya Murali
 
An Experimentation Framework: How to Position for Triple Digit Growth
An Experimentation Framework: How to Position for Triple Digit GrowthAn Experimentation Framework: How to Position for Triple Digit Growth
An Experimentation Framework: How to Position for Triple Digit GrowthOptimizely
 
Recent Trends in Personalization at Netflix
Recent Trends in Personalization at NetflixRecent Trends in Personalization at Netflix
Recent Trends in Personalization at NetflixJustin Basilico
 
Startupfest 2015: KRISHNA GADE (Pinterest) - "How To" Stage
Startupfest 2015: KRISHNA GADE (Pinterest) - "How To" StageStartupfest 2015: KRISHNA GADE (Pinterest) - "How To" Stage
Startupfest 2015: KRISHNA GADE (Pinterest) - "How To" StageStartupfest
 
Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...
Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...
Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...DevOpsDays Tel Aviv
 

Contenu connexe

Tendances

4 Steps Toward Scientific A/B Testing
4 Steps Toward Scientific A/B Testing4 Steps Toward Scientific A/B Testing
4 Steps Toward Scientific A/B TestingJanessa Lantz
 
SXSW 2016 - Everything you think about A/B testing is wrong
SXSW 2016 - Everything you think about A/B testing is wrongSXSW 2016 - Everything you think about A/B testing is wrong
SXSW 2016 - Everything you think about A/B testing is wrongDan Chuparkoff
 
Practical Introduction to A/B Testing
Practical Introduction to A/B TestingPractical Introduction to A/B Testing
Practical Introduction to A/B TestingAlex Alwan
 
A/B Testing Best Practices - Do's and Don'ts
A/B Testing Best Practices - Do's and Don'tsA/B Testing Best Practices - Do's and Don'ts
A/B Testing Best Practices - Do's and Don'tsRamkumar Ravichandran
 
A/B Testing at Pinterest: Building a Culture of Experimentation
A/B Testing at Pinterest: Building a Culture of Experimentation A/B Testing at Pinterest: Building a Culture of Experimentation
A/B Testing at Pinterest: Building a Culture of Experimentation WrangleConf
 
A/B testing in Firebase. Intermediate and advanced approach
A/B testing in Firebase. Intermediate and advanced approachA/B testing in Firebase. Intermediate and advanced approach
A/B testing in Firebase. Intermediate and advanced approachGameCamp
 
10 Guidelines for A/B Testing
10 Guidelines for A/B Testing10 Guidelines for A/B Testing
10 Guidelines for A/B TestingEmily Robinson
 
The Scientific Method of Experimentation by Google PM
The Scientific Method of Experimentation by Google PMThe Scientific Method of Experimentation by Google PM
The Scientific Method of Experimentation by Google PMProduct School
 
Music Personalization At Spotify
Music Personalization At SpotifyMusic Personalization At Spotify
Music Personalization At SpotifyVidhya Murali
 
Intro to A/B Testing by Ever's Senior Product Manager
Intro to A/B Testing by Ever's Senior Product ManagerIntro to A/B Testing by Ever's Senior Product Manager
Intro to A/B Testing by Ever's Senior Product ManagerProduct School
 
AB Testing at Expedia
AB Testing at ExpediaAB Testing at Expedia
AB Testing at ExpediaPaul Lucas
 
Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.js
Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.jsNetflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.js
Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.jsChris Saint-Amant
 
Machine Learning and Big Data for Music Discovery at Spotify
Machine Learning and Big Data for Music Discovery at SpotifyMachine Learning and Big Data for Music Discovery at Spotify
Machine Learning and Big Data for Music Discovery at SpotifyChing-Wei Chen
 
Building an A/B Testing Analytics System with R and Shiny
Building an A/B Testing Analytics System with R and ShinyBuilding an A/B Testing Analytics System with R and Shiny
Building an A/B Testing Analytics System with R and ShinyEmily Robinson
 
Netflix talk at ML Platform meetup Sep 2019
Netflix talk at ML Platform meetup Sep 2019Netflix talk at ML Platform meetup Sep 2019
Netflix talk at ML Platform meetup Sep 2019Faisal Siddiqi
 
The evolution of the big data platform @ Netflix (OSCON 2015)
The evolution of the big data platform @ Netflix (OSCON 2015)The evolution of the big data platform @ Netflix (OSCON 2015)
The evolution of the big data platform @ Netflix (OSCON 2015)Eva Tse
 
Building Data Pipelines for Music Recommendations at Spotify
Building Data Pipelines for Music Recommendations at SpotifyBuilding Data Pipelines for Music Recommendations at Spotify
Building Data Pipelines for Music Recommendations at SpotifyVidhya Murali
 
An Experimentation Framework: How to Position for Triple Digit Growth
An Experimentation Framework: How to Position for Triple Digit GrowthAn Experimentation Framework: How to Position for Triple Digit Growth
An Experimentation Framework: How to Position for Triple Digit GrowthOptimizely
 
Recent Trends in Personalization at Netflix
Recent Trends in Personalization at NetflixRecent Trends in Personalization at Netflix
Recent Trends in Personalization at NetflixJustin Basilico
 

Tendances (20)

4 Steps Toward Scientific A/B Testing
4 Steps Toward Scientific A/B Testing4 Steps Toward Scientific A/B Testing
4 Steps Toward Scientific A/B Testing
 
SXSW 2016 - Everything you think about A/B testing is wrong
SXSW 2016 - Everything you think about A/B testing is wrongSXSW 2016 - Everything you think about A/B testing is wrong
SXSW 2016 - Everything you think about A/B testing is wrong
 
Practical Introduction to A/B Testing
Practical Introduction to A/B TestingPractical Introduction to A/B Testing
Practical Introduction to A/B Testing
 
A/B Testing Best Practices - Do's and Don'ts
A/B Testing Best Practices - Do's and Don'tsA/B Testing Best Practices - Do's and Don'ts
A/B Testing Best Practices - Do's and Don'ts
 
A/B Testing at Pinterest: Building a Culture of Experimentation
A/B Testing at Pinterest: Building a Culture of Experimentation A/B Testing at Pinterest: Building a Culture of Experimentation
A/B Testing at Pinterest: Building a Culture of Experimentation
 
A/B testing in Firebase. Intermediate and advanced approach
A/B testing in Firebase. Intermediate and advanced approachA/B testing in Firebase. Intermediate and advanced approach
A/B testing in Firebase. Intermediate and advanced approach
 
10 Guidelines for A/B Testing
10 Guidelines for A/B Testing10 Guidelines for A/B Testing
10 Guidelines for A/B Testing
 
The Scientific Method of Experimentation by Google PM
The Scientific Method of Experimentation by Google PMThe Scientific Method of Experimentation by Google PM
The Scientific Method of Experimentation by Google PM
 
Music Personalization At Spotify
Music Personalization At SpotifyMusic Personalization At Spotify
Music Personalization At Spotify
 
Intro to A/B Testing by Ever's Senior Product Manager
Intro to A/B Testing by Ever's Senior Product ManagerIntro to A/B Testing by Ever's Senior Product Manager
Intro to A/B Testing by Ever's Senior Product Manager
 
AB Testing at Expedia
AB Testing at ExpediaAB Testing at Expedia
AB Testing at Expedia
 
Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.js
Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.jsNetflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.js
Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.js
 
Recommending and searching @ Spotify
Recommending and searching @ SpotifyRecommending and searching @ Spotify
Recommending and searching @ Spotify
 
Machine Learning and Big Data for Music Discovery at Spotify
Machine Learning and Big Data for Music Discovery at SpotifyMachine Learning and Big Data for Music Discovery at Spotify
Machine Learning and Big Data for Music Discovery at Spotify
 
Building an A/B Testing Analytics System with R and Shiny
Building an A/B Testing Analytics System with R and ShinyBuilding an A/B Testing Analytics System with R and Shiny
Building an A/B Testing Analytics System with R and Shiny
 
Netflix talk at ML Platform meetup Sep 2019
Netflix talk at ML Platform meetup Sep 2019Netflix talk at ML Platform meetup Sep 2019
Netflix talk at ML Platform meetup Sep 2019
 
The evolution of the big data platform @ Netflix (OSCON 2015)
The evolution of the big data platform @ Netflix (OSCON 2015)The evolution of the big data platform @ Netflix (OSCON 2015)
The evolution of the big data platform @ Netflix (OSCON 2015)
 
Building Data Pipelines for Music Recommendations at Spotify
Building Data Pipelines for Music Recommendations at SpotifyBuilding Data Pipelines for Music Recommendations at Spotify
Building Data Pipelines for Music Recommendations at Spotify
 
An Experimentation Framework: How to Position for Triple Digit Growth
An Experimentation Framework: How to Position for Triple Digit GrowthAn Experimentation Framework: How to Position for Triple Digit Growth
An Experimentation Framework: How to Position for Triple Digit Growth
 
Recent Trends in Personalization at Netflix
Recent Trends in Personalization at NetflixRecent Trends in Personalization at Netflix
Recent Trends in Personalization at Netflix
 

Similaire à Why everything is an A/B Test at Pinterest

Startupfest 2015: KRISHNA GADE (Pinterest) - "How To" Stage
Startupfest 2015: KRISHNA GADE (Pinterest) - "How To" StageStartupfest 2015: KRISHNA GADE (Pinterest) - "How To" Stage
Startupfest 2015: KRISHNA GADE (Pinterest) - "How To" StageStartupfest
 
Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...
Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...
Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...DevOpsDays Tel Aviv
 
Software Release Orchestration and the Enterprise
Software Release Orchestration and the EnterpriseSoftware Release Orchestration and the Enterprise
Software Release Orchestration and the EnterpriseXebiaLabs
 
Building turn-key recommendations for 5% of internet video
Building turn-key recommendations for 5% of internet videoBuilding turn-key recommendations for 5% of internet video
Building turn-key recommendations for 5% of internet videoNir Yungster
 
SFScon21 - Paolo d’Incau - Going to production in a few months – How we did it!
SFScon21 - Paolo d’Incau - Going to production in a few months – How we did it!SFScon21 - Paolo d’Incau - Going to production in a few months – How we did it!
SFScon21 - Paolo d’Incau - Going to production in a few months – How we did it!South Tyrol Free Software Conference
 
Video Recommendation Engines as a Service
Video Recommendation Engines as a ServiceVideo Recommendation Engines as a Service
Video Recommendation Engines as a ServiceKamil Sindi
 
Making the Switch from HP Quality Center to qTest
Making the Switch from HP Quality Center to qTestMaking the Switch from HP Quality Center to qTest
Making the Switch from HP Quality Center to qTestQASymphony
 
Developing Quality Products Quickly through a Culture of CI/CD
Developing Quality Products Quickly through a Culture of CI/CDDeveloping Quality Products Quickly through a Culture of CI/CD
Developing Quality Products Quickly through a Culture of CI/CDPaul Hepworth
 
Monktoberfest Fast Delivery
Monktoberfest Fast DeliveryMonktoberfest Fast Delivery
Monktoberfest Fast DeliveryAdrian Cockcroft
 
Continuous testing for Agile and DevOps teams
Continuous testing for Agile and DevOps teamsContinuous testing for Agile and DevOps teams
Continuous testing for Agile and DevOps teamsLaurent PY
 
The Firebase tier for your mobile app - DevFest CH
The Firebase tier for your mobile app - DevFest CHThe Firebase tier for your mobile app - DevFest CH
The Firebase tier for your mobile app - DevFest CHMatteo Bonifazi
 
Mobile video ads: How to acquire high quality new users and measure marketing...
Mobile video ads: How to acquire high quality new users and measure marketing...Mobile video ads: How to acquire high quality new users and measure marketing...
Mobile video ads: How to acquire high quality new users and measure marketing...Adjust
 
Continuous Delivery at Snyk
Continuous Delivery at SnykContinuous Delivery at Snyk
Continuous Delivery at SnykAnton Drukh
 
Data-Driven Software Testing: The New, Lean Approach to Quality
Data-Driven Software Testing: The New, Lean Approach to QualityData-Driven Software Testing: The New, Lean Approach to Quality
Data-Driven Software Testing: The New, Lean Approach to QualityTechWell
 
Freedom and Responsibility
Freedom and ResponsibilityFreedom and Responsibility
Freedom and ResponsibilityMike Ruangutai
 
Enabling Business Outcomes Through Extreme Programming and Continuous Delivery
Enabling Business Outcomes Through Extreme Programming and Continuous DeliveryEnabling Business Outcomes Through Extreme Programming and Continuous Delivery
Enabling Business Outcomes Through Extreme Programming and Continuous DeliveryVMware Tanzu
 
Intro to Data Analytics with Oscar's Director of Product
Intro to Data Analytics with Oscar's Director of Product Intro to Data Analytics with Oscar's Director of Product
Intro to Data Analytics with Oscar's Director of ProductProduct School
 
How to make sure the right quality is delivered by my translation vendor? (Se...
How to make sure the right quality is delivered by my translation vendor? (Se...How to make sure the right quality is delivered by my translation vendor? (Se...
How to make sure the right quality is delivered by my translation vendor? (Se...TAUS - The Language Data Network
 
From four to forty in four years - lessons from growing a team
From four to forty in four years - lessons from growing a teamFrom four to forty in four years - lessons from growing a team
From four to forty in four years - lessons from growing a teamRich Allen
 
Developing and deploying the Netflix API service
Developing and deploying the Netflix API serviceDeveloping and deploying the Netflix API service
Developing and deploying the Netflix API serviceSangeeta Narayanan
 

Similaire à Why everything is an A/B Test at Pinterest (20)

Startupfest 2015: KRISHNA GADE (Pinterest) - "How To" Stage
Startupfest 2015: KRISHNA GADE (Pinterest) - "How To" StageStartupfest 2015: KRISHNA GADE (Pinterest) - "How To" Stage
Startupfest 2015: KRISHNA GADE (Pinterest) - "How To" Stage
 
Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...
Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...
Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...
 
Software Release Orchestration and the Enterprise
Software Release Orchestration and the EnterpriseSoftware Release Orchestration and the Enterprise
Software Release Orchestration and the Enterprise
 
Building turn-key recommendations for 5% of internet video
Building turn-key recommendations for 5% of internet videoBuilding turn-key recommendations for 5% of internet video
Building turn-key recommendations for 5% of internet video
 
SFScon21 - Paolo d’Incau - Going to production in a few months – How we did it!
SFScon21 - Paolo d’Incau - Going to production in a few months – How we did it!SFScon21 - Paolo d’Incau - Going to production in a few months – How we did it!
SFScon21 - Paolo d’Incau - Going to production in a few months – How we did it!
 
Video Recommendation Engines as a Service
Video Recommendation Engines as a ServiceVideo Recommendation Engines as a Service
Video Recommendation Engines as a Service
 
Making the Switch from HP Quality Center to qTest
Making the Switch from HP Quality Center to qTestMaking the Switch from HP Quality Center to qTest
Making the Switch from HP Quality Center to qTest
 
Developing Quality Products Quickly through a Culture of CI/CD
Developing Quality Products Quickly through a Culture of CI/CDDeveloping Quality Products Quickly through a Culture of CI/CD
Developing Quality Products Quickly through a Culture of CI/CD
 
Monktoberfest Fast Delivery
Monktoberfest Fast DeliveryMonktoberfest Fast Delivery
Monktoberfest Fast Delivery
 
Continuous testing for Agile and DevOps teams
Continuous testing for Agile and DevOps teamsContinuous testing for Agile and DevOps teams
Continuous testing for Agile and DevOps teams
 
The Firebase tier for your mobile app - DevFest CH
The Firebase tier for your mobile app - DevFest CHThe Firebase tier for your mobile app - DevFest CH
The Firebase tier for your mobile app - DevFest CH
 
Mobile video ads: How to acquire high quality new users and measure marketing...
Mobile video ads: How to acquire high quality new users and measure marketing...Mobile video ads: How to acquire high quality new users and measure marketing...
Mobile video ads: How to acquire high quality new users and measure marketing...
 
Continuous Delivery at Snyk
Continuous Delivery at SnykContinuous Delivery at Snyk
Continuous Delivery at Snyk
 
Data-Driven Software Testing: The New, Lean Approach to Quality
Data-Driven Software Testing: The New, Lean Approach to QualityData-Driven Software Testing: The New, Lean Approach to Quality
Data-Driven Software Testing: The New, Lean Approach to Quality
 
Freedom and Responsibility
Freedom and ResponsibilityFreedom and Responsibility
Freedom and Responsibility
 
Enabling Business Outcomes Through Extreme Programming and Continuous Delivery
Enabling Business Outcomes Through Extreme Programming and Continuous DeliveryEnabling Business Outcomes Through Extreme Programming and Continuous Delivery
Enabling Business Outcomes Through Extreme Programming and Continuous Delivery
 
Intro to Data Analytics with Oscar's Director of Product
Intro to Data Analytics with Oscar's Director of Product Intro to Data Analytics with Oscar's Director of Product
Intro to Data Analytics with Oscar's Director of Product
 
How to make sure the right quality is delivered by my translation vendor? (Se...
How to make sure the right quality is delivered by my translation vendor? (Se...How to make sure the right quality is delivered by my translation vendor? (Se...
How to make sure the right quality is delivered by my translation vendor? (Se...
 
From four to forty in four years - lessons from growing a team
From four to forty in four years - lessons from growing a teamFrom four to forty in four years - lessons from growing a team
From four to forty in four years - lessons from growing a team
 
Developing and deploying the Netflix API service
Developing and deploying the Netflix API serviceDeveloping and deploying the Netflix API service
Developing and deploying the Netflix API service
 

Dernier

Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Boston Institute of Analytics
 
Role of Consumer Insights in business transformation
Role of Consumer Insights in business transformationRole of Consumer Insights in business transformation
Role of Consumer Insights in business transformationAnnie Melnic
 
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdfEnglish-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdfblazblazml
 
Digital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing worksDigital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing worksdeepakthakur548787
 
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...Jack Cole
 
Statistics For Management by Richard I. Levin 8ed.pdf
Statistics For Management by Richard I. Levin 8ed.pdfStatistics For Management by Richard I. Levin 8ed.pdf
Statistics For Management by Richard I. Levin 8ed.pdfnikeshsingh56
 
Non Text Magic Studio Magic Design for Presentations L&P.pdf
Non Text Magic Studio Magic Design for Presentations L&P.pdfNon Text Magic Studio Magic Design for Presentations L&P.pdf
Non Text Magic Studio Magic Design for Presentations L&P.pdfPratikPatil591646
 
DATA ANALYSIS using various data sets like shoping data set etc
DATA ANALYSIS using various data sets like shoping data set etcDATA ANALYSIS using various data sets like shoping data set etc
DATA ANALYSIS using various data sets like shoping data set etclalithasri22
 
Presentation of project of business person who are success
Presentation of project of business person who are successPresentation of project of business person who are success
Presentation of project of business person who are successPratikSingh115843
 
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis model
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis modelDecoding Movie Sentiments: Analyzing Reviews with Data Analysis model
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis modelBoston Institute of Analytics
 
Digital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdfDigital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdfNicoChristianSunaryo
 
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...Dr Arash Najmaei ( Phd., MBA, BSc)
 
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBoston Institute of Analytics
 
IBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaIBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaManalVerma4
 

Dernier (17)

Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
 
Role of Consumer Insights in business transformation
Role of Consumer Insights in business transformationRole of Consumer Insights in business transformation
Role of Consumer Insights in business transformation
 
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdfEnglish-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
 
Digital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing worksDigital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing works
 
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
 
Data Analysis Project: Stroke Prediction
Data Analysis Project: Stroke PredictionData Analysis Project: Stroke Prediction
Data Analysis Project: Stroke Prediction
 
Statistics For Management by Richard I. Levin 8ed.pdf
Statistics For Management by Richard I. Levin 8ed.pdfStatistics For Management by Richard I. Levin 8ed.pdf
Statistics For Management by Richard I. Levin 8ed.pdf
 
Non Text Magic Studio Magic Design for Presentations L&P.pdf
Non Text Magic Studio Magic Design for Presentations L&P.pdfNon Text Magic Studio Magic Design for Presentations L&P.pdf
Non Text Magic Studio Magic Design for Presentations L&P.pdf
 
DATA ANALYSIS using various data sets like shoping data set etc
DATA ANALYSIS using various data sets like shoping data set etcDATA ANALYSIS using various data sets like shoping data set etc
DATA ANALYSIS using various data sets like shoping data set etc
 
2023 Survey Shows Dip in High School E-Cigarette Use
2023 Survey Shows Dip in High School E-Cigarette Use2023 Survey Shows Dip in High School E-Cigarette Use
2023 Survey Shows Dip in High School E-Cigarette Use
 
Presentation of project of business person who are success
Presentation of project of business person who are successPresentation of project of business person who are success
Presentation of project of business person who are success
 
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis model
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis modelDecoding Movie Sentiments: Analyzing Reviews with Data Analysis model
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis model
 
Digital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdfDigital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdf
 
Insurance Churn Prediction Data Analysis Project
Insurance Churn Prediction Data Analysis ProjectInsurance Churn Prediction Data Analysis Project
Insurance Churn Prediction Data Analysis Project
 
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...
 
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
 
IBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaIBEF report on the Insurance market in India
IBEF report on the Insurance market in India
 

Why everything is an A/B Test at Pinterest