Stream processing in Mercari - Devsumi 2015 autumn LT

1. Stream processing
in Mercari
Developers Summit 2015 Autumn LT
@kazeburo Masahiro Nagano

2. Me
•長野雅広(Masahiro Nagano)
•@kazeburo
•Mercari, Inc.
•Principal Engineer, Site Reliability

3. Your Friendly Mobile
Marketplace

4. Stream processing
for Monitoring

5. Monitoring

6. collect metrics value
• Just get current value/point
• e.g. load avg, memory usage
• Get counter/incremental value and calculate
changes
• e.g. traffic, cpu usage
• Parse log and count value by your self
• e.g. access_log, error_log, custom_log

7. Log monitoring
Past and present

8. While ago
• Make a script for parsing and count logs
• Execute periodically from monitoring tool
• Too many tail (io) and grep (cpu) (;_;)
#!/bin/sh
set -e
LOG_WATCH=20000
TARGET=$(date -d '61 second ago' +%H:%M:[0-9][0-9])
tail -$LOG_WATCH /var/log/httpd/service_access_log | grep "$TARGET" |
wc -l
tail -$LOG_WATCH /var/log/httpd/service_access_log | grep "$TARGET" |
grep '" 500 ' wc -l

9. fluentd + datacounter
Web
access_log
in_tail
Web
access_log
Web
access_log
aggregate
datacounter
<match mercari.access_log>
type datacounter
count_interval 1m
count_key status
pattern1 2xx ^2dd$
pattern2 3xx ^3dd$
pattern4 4xx ^4dd$
pattern5 5xx ^5dd$
tag status.datacount.web
</match>
<match status.datacount.web>
type zabbix
zabbix_server 10.x.x.x
</match>

10. Cons of fluentd
+datacounter
•Gigantic fluend.conf
•need a configuration file generator?
•Write fluend plugin for responding
various needs?
•restart fluentd is required for adding a
new tabulation

12. Norikra
•Norikra is an awesome open source
product by Mr. tagomori, provides
“Stream processing” with SQL
•We LOVE SQL!!

13. 219.109.xx.xx - - [09/Nov/2014:06:50:07 +0900] "GET /mt.js HTTP/1.1" 200 6339
"http://blog.nomadscafe.jp/2013/06/webdb-pressvol172.html" "Mozilla/5.0
(Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
50.116.xx.xx - - [09/Nov/2014:06:54:41 +0900] "GET /2014/09/line-
isucon4-51192.html HTTP/1.1" 200 30194 "http://www.google.co.uk/url?
sa=t&source=web&cd=1" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2
(KHTML, like Gecko) Chrome/15.0.874.92 Safari/535.2"
76.164.xx.xx - - [09/Nov/2014:07:09:21 +0900] "GET / HTTP/1.0" 200 75606
"http://blog.nomadscafe.jp/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/
537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
SELECT COUNT(1, status like “5%”) AS count_5x,
COUNT(1, status like “2%”) AS count_2x
FROM access_log.win:time_batch(1 min)
{ “count_2x”: 765, “count_5x”: 2 }
Norikra

14. After Norika
Web
access_log
in_tail
Web
access_log
Web
access_log
aggregate
SQL投入
Norikra

15. Pros of Norikra
• Do not need a gigantic configuration file
• Do not required restarting a daemon
• Schema less
• analysis any log data like error_log,
application specific log
• SQL
• every engineer/producer can write SQL

16. Graph/Alert
•zabbix
•nagios
•GrowthForecast (graph only)
•Data Dog
•mackerel

18. Worker/Batch
WebWeb
Stream Processing in Mercari
Web/App
access_log
Worker/Batch
create SQL
Norikra
error_log
app_log
error_log
app_log
define Alert
Mackerel
Generates Graph
automatically!!

19. Alternatives and Competitors
• PipelineDB
• www.pipelinedb.com
• Azure Stream Analytics
• azure.microsoft.com/ja-jp/services/stream-
analytics/
• Amazon Kinesis Analytics(coming soon)
• aws.amazon.com/kinesis/analytics/

20. end