Pinterest moved 100TB of data from MySQL databases to S3 and Hadoop continuously using a new data pipeline. The pipeline uses Kafka to stream database change events in real-time. It incorporates periodic compaction to merge snapshots and deltas into a compact format with 15 minute latency. The new system provides reliability, scalability, and enables features like real-time search and recommendations.

2. Movingthe
needleofthePin:
Oct, 2018Henry Cai
www.linkedin.com/in/hecai
Streaming100TBofpinsfrom
MySQLtoS3/Hadoop
continuously@Pinterest

3. Pinterestisthe
visualdiscovery
engine.Mission
Helppeoplediscoveranddowhattheylove.

4. >250M
80%
75%
ofsignupsare
fromoutside
theU.S.
ofPinnersuse
Pinterestfrom
mobile
100B
Pinsand
2BBoards
monthly
activeusers

5. Data-driven
products
• Personalized
recommendation
• SpamControl
• SearchQuality
• A/BExperiments
• RelatedPins
• …

6. DataPipeline
stats
• >1PBdata/day
• >10Mmessages/second
• >800Bmessages/day
• >2,000kafkabrokers
• >50,000clienthosts

7. Dataingestion
types
• Onlinelogging
• Databasesnapshots

8. 2016
pipeline

9. Dataingestion@Pinterest 2016
Pinterest Services
Singer
Kafka

10. Dataingestion@Pinterest 2016
Pinterest Services
Singer
Kafka
events

11. Dataingestion@Pinterest 2016
Pinterest Services
Singer
Kafka
events
Real-time
consumers
Merced
Tracker

12. Dataingestion@Pinterest 2016
Pinterest Services
Singer
Kafka
events
Real-time
consumers
Databases
Merced
Tracker

13. Dataingestion@Pinterest 2016
Pinterest Services
Singer
Kafka
events
Real-time
consumers
Databases
Merced
Tracker

14. Dataingestion@Pinterest 2016
Pinterest Services
Singer
Kafka
events
Real-time
consumers
Databases Logical backup
Merced
Tracker

15. DBingestion@Pinterest
Version1
DatabasesShard1 Slave
Shard1 DrSlave
Shard1 Master
Mysqldump
Hadoop
Streaming
Mapper1
Shard2 Slave
Shard2 DrSlave
Shard2 Master
Mysqldump Hadoop
Streaming
Mapper2

16. DBingestion@Pinterest
Version2
Databases logical csv
backup
Tracker
Version1
Shard1 Slave
Shard1 DrSlave
Shard1 Master
Mysqldump
Hadoop
Streaming
Mapper1
Shard2 Slave
Shard2 DrSlave
Shard2 Master
Mysqldump Hadoop
Streaming
Mapper2

17. Painpoints
Constraints
• Reliabilitycausedbymysqlhostshiccup
• Pullingover100TBdatadailybutonlyafewTB
changedeveryday
• Longlatency>24hour
Future:DBChangeStreams
• Trulycapturesdbtransactions
• Across-regioncacheinvalidation
• Realtimesearchindexbuilding
• RealtimeRecommendationEngine

18. The
newpipeline

19. Dataingestion@Pinterest now
Pinterest Services
Singer
Kafka
events

20. Dataingestion@Pinterest now
Pinterest Services
Singer
Kafka
events
Databases
DB/Kafka
Bridge

21. Dataingestion@Pinterest now
Pinterest Services
Singer
Kafka
events
Databases
DB/Kafka
Bridge
Merced

22. Dataingestion@Pinterest now
Pinterest Services
Singer
Kafka
events
Databases
DB/Kafka
Bridge
Merced
Watermill

23. Dataingestion@Pinterest now
Pinterest Services
Singer
Kafka
events
Real-time
consumers
Databases
DB/Kafka
Bridge
Merced
Watermill

24. DB/KafkaBridge(Maxwell)
Pinterest Services
Singer
Kafka
events
Real-time
consumers
Databases
Merced
Watermill
DB/Kafka
Bridge

25. DB/KafkaBridge
Replica-SetNode
Maxwell_position
Maxwell_schema
MySQL Processes and Schemas
Maxwell Tables
Binlog File
Shard1 Shard2 Shard3
User Tables

26. DB/KafkaBridge
Replica-SetNode
Maxwell_position
Maxwell_schema
MySQL Processes and Schemas
Maxwell Tables
MySQL Processes (Co-located with MySQL Process)
Binlog File
Shard1 Shard2 Shard3
User Tables
Kafka User Topic
Kafka Pin Topic
BinLog
Tailer
Thread
InMemory
Queue
Async
Kafka
Producer
Thread
• BasedonMaxwell/Binlog-Connector
• AddGTIDsupport
• Addhandlingforretry/out-of-ordermessages
• Co-locatewithmysql
• Listensonmaster/slave

27. DB/Kafka
Bridge
Watermillcompaction
Pinterest Services
Singer
Kafka
events
Real-time
consumers
Databases
Merced
Watermill

28. Compaction
ForOneShard
• HashJoinbetweensnapshotanddelta
• Deltaloadedinmemoryfirstassidelookup
• Basesnapshotwaspipedthroughthemappernodeand
compareagainstlookuptable
- Lookupfail,snapshotrecordemittooutput
- Lookupsucceed,butsnapshotrecordold,skipthe
snapshot
- Lookupsucceed,butsnapshotrecordnewer,remove
lookuprecord
• Attheend,appendtheremaininglookuprecordstooutput
Delta Shard 1
Old Snapshot
Shard 1
Compactor
New Snapshot
Shard 1

29. IncrementalDBingestionsequence
MySQL
Maxwell
Kafka

30. IncrementalDBingestionsequence
MySQL
Maxwell Merced
Delta
Kafka

31. IncrementalDBingestionsequence
MySQL
Maxwell Merced Periodic
Compaction
Snapshot1
Delta
Snapshot2
Kafka

32. IncrementalDBingestionsequence
MySQL
Tracker
Batch
Backup
Backup
Snapshot
Maxwell Merced Periodic
Compaction
Snapshot1
Delta
Snapshot2
Bootstrapper
Kafka

33. IncrementalDBingestionsequence
MySQL
Tracker
Batch
Backup
Backup
Snapshot
Maxwell Merced Periodic
Compaction
Periodic
FileGC
Snapshot1
Delta
Snapshot2
Differ
Bootstrapper
Kafka

34. IncrementalDBingestionsequence
MySQL
Tracker
Batch
Backup
Maxwell Merced Periodic
Compaction
Periodic
FileGC
SELECT
FROM
rt_users
Snapshot1
Delta
Snapshot2
Custom
Input
Format
Differ
Bootstrapper
Backup
Snapshot
Kafka

35. DataLifecycleandTimelineManagement
DailyDump
11:30
Bootstrap
Snapshot
11:55
1
1
:
3
0
1
1
:
5
5
Timeline

36. DataLifecycleandTimelineManagement
Merced Delta
12:01
DailyDump
11:30
Bootstrap
Snapshot
11:55
1
1
:
3
0
1
1
:
5
5
1
2
:
0
1
Kafka
Timeline

37. DataLifecycleandTimelineManagement
Merced CompactionDelta
12:01
Snapshot
12:10AM
DailyDump
11:30
Bootstrap
Snapshot
11:55
1
1
:
3
0
1
1
:
5
5
1
2
:
0
1
1
2
:
1
0
Kafka
Timeline

38. DataLifecycleandTimelineManagement
Merced CompactionDelta
12:01
Snapshot
12:10AM
12:15
Select
DailyDump
11:30
Bootstrap
Snapshot
11:55
1
1
:
3
0
1
1
:
5
5
1
2
:
0
1
1
2
:
1
0
Kafka
Timeline

39. DataLifecycleandTimelineManagement
Merced CompactionDelta
12:01
Snapshot
12:10AM
12:15
Select
DailyDump
11:30
Bootstrap
Snapshot
11:55
1
1
:
3
0
1
1
:
5
5
1
2
:
0
1
1
2
:
1
0
Kafka
Timeline
ProcessedUpTo
CurrentSnapshot

40. DataLifecycleandTimelineManagement
Merced CompactionDelta
12:01
Snapshot
12:10AM
12:15
Select
DailyDump
11:30
Bootstrap
Snapshot
11:55
DailyDump
11:45
Bootstrap
Snapshot
12:20
1
1
:
3
0
1
1
:
5
5
1
2
:
0
1
1
2
:
1
0
1
1
:
4
5
1
2
:
2
0
Kafka
Timeline

41. DataLifecycleandTimelineManagement
Merced CompactionDelta
12:01
Snapshot
12:10AM
12:15
Select
DailyDump
11:30
Bootstrap
Snapshot
11:55
DailyDump
11:45
Bootstrap
Snapshot
12:20
12:25
Select
1
1
:
3
0
1
1
:
5
5
1
2
:
0
1
1
2
:
1
0
1
1
:
4
5
1
2
:
2
0
Kafka
Timeline

42. DataLifecycleandTimelineManagement
Merced CompactionDelta
12:01
Snapshot
12:10AM
12:15
Select
DailyDump
11:30
Bootstrap
Snapshot
11:55
DailyDump
11:45
Bootstrap
Snapshot
12:20
12:25
Select
1
1
:
3
0
1
1
:
5
5
1
2
:
0
1
1
2
:
1
0
1
1
:
4
5
1
2
:
2
0
Kafka
Timeline
CurrentSnapshot
ProcessedUpto

43. DataLifecycleandTimelineManagement
1
1
:
3
0
1
1
:
5
5
1
2
:
0
1
1
2
:
1
0
1
1
:
4
5
1
2
:
2
0
Timeline
Merced CompactionDelta
12:01
Snapshot
12:10AM
12:15
Select
DailyDump
11:30
Bootstrap
Snapshot
11:55
DailyDump
11:45
Bootstrap
Snapshot
12:20
12:25
Select
Kafka
ProcessedUpto
… …NextCompaction……

44. DataLifecycleandTimelineManagement
Merced CompactionDelta
12:01
Snapshot
12:10AM
DailyDump
11:30
Bootstrap
Snapshot
11:55
DailyDump
11:45
Bootstrap
Snapshot
12:20
1
1
:
3
0
1
1
:
5
5
1
2
:
0
1
1
2
:
1
0
1
1
:
4
5
1
2
:
2
0
Kafka
Timeline
CurrentSnapshot
Periodic
GC

45. DataLifecycleandTimelineManagement
Merced CompactionDelta
12:01
Snapshot
12:10AM
DailyDump
11:30
Bootstrap
Snapshot
11:55
DailyDump
11:45
Bootstrap
Snapshot
12:20
1
1
:
3
0
1
1
:
5
5
1
2
:
0
1
1
2
:
1
0
Kafka
Timeline
PossibleRewind
Periodic
GC

46. Consistency
• MySQLMaster/SlaveFailover,ShardMigration
• MySQLTransactions:
• Splitbetweentables,splitbetweenKafkamessages
• Ordering
• BetweenINSERTandUPDATE
• BetweenUPDATEandDELETE
• SoftDELETEvs.HardDELETE
• Consistencybetweenmultiplebootstrapand
incrementalstreams
• DuplicateRecords

47. Scalability
• Partitioning
• ShardedMySQL
- Shardbaseddbsnapshotanddeltafiles
- Twolevelsharinginthecasethatoriginalshardsarenot
balanced
• UnShardeddataset
- Usehash+modtopartitionthedataonbothsnapshot
anddeltafile
• Filefilteringusingpredicatepushdown:
• Onshard/partitionlevel
• OnS3directory,fileandrecordlevel
10X

48. KafkaNuances
• MessageOrdering:
• Asyncproducerbutstillneedtomaintainmessageorder
• MaintainorderbetweenS3fileandwithinS3file
• At-least-oncedelivery
• Duplicatemessages
• MySQLGTIDnotalwaysincreasing
• DealwithKafkaclusterhiccup:
• produceracks=2
• cleanleaderelection

49. S3Nuances
● Eventual Consistency
● Read-after-write is OK, but not PUT followed
by LIST
● Directory listing is slow
● Shorter SLA —> More smaller files
● In early iterations, directly listing >> file content
reading
● Rate Limit:
● Launching thousands of mappers would
quickly hit S3 rate limit

50. PIIProcessing
• username,emailaddressetcneedstobe
filteredout
• ipaddressneedstobefilteredout
john.doe@abc.com
Justin Bieber
192.168.0.1

51. PIIProcessing
• username,emailaddressetcneedstobe
filteredout
• ipaddressneedstobefilteredout
john.doe@abc.com
Justin Bieber
192.168.0.1
ColumnarLayout
andIncremental
Processing
• Useparquetformattosupportfastquerieson
subsetofcolumns
• ingest_timeasnewcolumntogetthe
incrementalresultsincethelastprocessing;

52. Operation

53. Bootstrap,synchronize&rewind
MySQL
Tracker
Batch
Backup
Backup
Snapshot
Maxwell Merced Periodic
Compaction
Snapshot1
Delta
Snapshot2
Bootstrapper
Kafka

54. Bootstrap,synchronize&rewind(cont)
• Wehavetheabilitytosynchronizeandrewind
• Incaseofsoftwarebugsornetworkglitches
• Snapshot(s)ontoBootstraptosynchronize
• AbilitytorewindviatheSnapshots/Bootstrapmechanism
MySQL
Tracker
Batch
Backup
Backup
Snapshot
Maxwell Merced Periodic
Compaction
Snapshot1
Delta
Snapshot2
Bootstrapper
Kafka

55. Schema
Management
andSchema
Change
• SchemaisUsedfor
• Identifytheprimarykeyoftherow
• Drivetheparquetfilegeneration
• DealingWithSchemachange
• Willissueanewbootstraponofflinetableschema
• Compactionwillstillusethesnapshotschema
(whichmightbeold)
ID C1 C2
123 …. …
124 … ….
125 …. …
126 … ….
dbname.table_name
new_column
….
…
….
…

56. Validation
• Validation
• CreatingcompactionbasedonfromandtoGTIDrange
• Compactionoutputvsbatchbackupoutput
• Monitoring
• Error,failure,stall
• Latencyoncompaction
Backup
Snapshot
Periodic
Compaction
Snapshot1 Snapshot2
Differ
Bootstrapper

57. Summary

58. Comparison
toother
technologies
• UberHudi(Hoodie)
• NotsupportingS3,OnlysupportJava8+,Avro

59. Comparison
toother
technologies
• UberHudi(Hoodie)
• NotsupportingS3,OnlysupportJava8+,Avro
• KafkaConnect
• Onlyingestion,nocompacting,synchronizebetween
bootstrap/incremental

60. Comparison
toother
technologies
• UberHudi(Hoodie)
• NotsupportingS3,OnlysupportJava8+,Avro
• KafkaConnect/Debezium
• Onlyingestion,nocompacting,synchronizebetween
bootstrap/incremental
• ApacheSqoop
• BasedonBatchMode

61. Takeaway
• Scalability
• support100TBofdatabasedata
• E2Elatencyof15minutes
• Reliability
• Strongdatabaseconsistencyonglobaltransactions,
messageordering,duplicatemessagehandling
• ValidationandMonitoring
• Operability
• Bootstrap,re-synchronize
• Schemamanagement

62. Futurework
• AdoptingKafkaExact-OnceProcessing
Model
• Kafkaasthedatabasechangestream
• Cacheinvalidationacrossdatacenters
• BuildingMaterializedViewsforMySQL
• GeneratingIncrementalRecommendationSignals
• OpenSource

63. Acknowledgement
• Jointworkfrommany
engineering,including
YuYang,ChunyanWang,
IndyPrentice,Shawn
Nguyen,YinianQi, and
manyothers

65. Thanks!

66. © Copyright, All Rights Reserved, Pinterest Inc. 2018