You are a developer, create applications that generate logs. You would like to monitor those logs to check what the application is doing in production. Or you are an operator in need for information about the whole platform. You need logs from the load balancer, proxy, database and the application. If possible you would like to correlate these logs as well. Maybe you are an analyst and you would like to create some graphs of the data you obtained. If one of these roles is you, the chance is big you heard about ELK. This is short for Elasticsearch, Logstash and Kibana. The goal for these projects is to obtain data (logstash), store it in a central repository (elasticsearch) to make it searchable and available for analysis. Having all this data is nice, but making it visible is even better, that is where Kibana comes in. With Kibana you can create nice dashboard giving insight into your data. ELK is a proven technology stack to handle your logs. During this talk I will present you the complete stack. I’ll show you how to import data with logstash, explain what happens in elasticsearch and create a dashboard using Kibana. I will also discuss some choices you have to make while storing the data, go into a number of possible architectures for the ELK stack. At the end you have a good idea about what ELK can do for you.

1. REALTIME DATA ANALYSIS
USING ELK
@jettroCoenradie

2. Jettro Coenradie
http://amsterdam.luminis.eu

3. Jettro Coenradie
http://amsterdam.luminis.eu

4. Jettro Coenradie
http://amsterdam.luminis.eu

5. Jettro Coenradie
http://amsterdam.luminis.eu

6. Jettro Coenradie
http://amsterdam.luminis.eu

7. REALTIME DATA ANALYSIS
USING ELK
Real time log analysis
Introduction of ELK components
Who is abusing my blog?
Lessons learned from IDMC project
Good to know

8. REALTIME DATA ANALYSIS
USING ELK
Real time log analysis
Introduction of ELK components
Who is abusing my blog?
Lessons learned from IDMC project
Good to know

9. REALTIME DATA ANALYSIS
USING ELK
Real time log analysis
Introduction of ELK components
Who is abusing my blog?
Lessons learned from IDMC project
Good to know

10. REALTIME DATA ANALYSIS
USING ELK
Real time log analysis
Introduction of ELK components
Who is abusing my blog?
Lessons learned from IDMC project
Good to know

11. REALTIME DATA ANALYSIS
USING ELK
Real time log analysis
Introduction of ELK components
Who is abusing my blog?
Lessons learned from IDMC project
Good to know

12. REALTIME LOG ANALYSIS

13. indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?js=wp-includes%2Fjs%2Fjquery%2Fjquery.js%3Fver%3D1.11.1%2Cwp-includes%2Fjs%2Fjquery%2Fjquery-migrate.min.js%3Fver%3D1.2.1%2Cwp-content%2Fthemes
%2Fspacious%2Fjs%2Fspacious-custom.js%3Fver%3D4.1.1 HTTP/1.1" 200 36442 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/akismet/_inc/form.js?ver=3.1.1 HTTP/1.1" 200 969 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css
%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3 HTTP/1.1" 200 36995 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/jetpack/modules/wpgroho.js?ver=4.1.1 HTTP/1.1" 200 1228 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X
10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/scripts/shBrushJScript.js?ver=3.0.9b HTTP/1.1" 200 2056 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0
(Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/scripts/shCore.js?ver=3.0.9b HTTP/1.1" 200 24190 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0
(Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-includes/js/comment-reply.min.js?ver=4.1.1 HTTP/1.1" 200 1026 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/themes/spacious/js/navigation.js?ver=4.1.1 HTTP/1.1" 200 1233 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X
10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/jetpack/modules/sharedaddy/sharing.js?ver=3.4.3 HTTP/1.1" 200 43124 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/styles/shCore.css?ver=3.0.9b HTTP/1.1" 200 7042 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0
(Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/styles/shThemeDefault.css?ver=3.0.9b HTTP/1.1" 200 3257 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/
5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-admin/admin-ajax.php?action=wordfence_logHuman&hid=57D2A62DA174F5DA747EC194B32CBD32&r=0.9168877548072487 HTTP/1.1" 200 461 "http://www.gridshore.nl/2014/07/26/transform-the-input-
before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/jetpack/_inc/genericons/genericons/svg HTTP/1.1" 404 32544 "http://www.gridshore.nl/wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver
%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:36 +0200] "GET /favicon.ico HTTP/1.1" 200 2829 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like
Gecko) Chrome/42.0.2311.90 Safari/537.36"
46.4.132.226 - - [21/Apr/2015:03:51:37 +0200] "GET /tag/united-states/feed/ HTTP/1.0" 200 15341 "http://www.gridshore.nl/tag/united-states/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
46.4.132.226 - - [21/Apr/2015:03:51:47 +0200] "GET /2008/11/04/threadsafe-applications-part-i/ HTTP/1.0" 200 57529 "http://www.gridshore.nl/2008/11/05/yes-we-canthe-day-after/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
46.4.132.226 - - [21/Apr/2015:03:52:01 +0200] "GET /2008/11/04/threadsafe-applications-part-i/feed/ HTTP/1.0" 200 7159 "http://www.gridshore.nl/2008/11/04/threadsafe-applications-part-i/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/
robot.html)"
54.197.215.119 - - [21/Apr/2015:03:52:10 +0200] "GET /favicon.ico HTTP/1.1" 200 2829 "-" "Pinterest/0.2 (+http://www.pinterest.com/)"
54.197.215.119 - - [21/Apr/2015:03:52:11 +0200] "GET /apple-touch-icon.png HTTP/1.1" 404 32507 "-" "Pinterest/0.2 (+http://www.pinterest.com/)"
46.4.132.226 - - [21/Apr/2015:03:52:12 +0200] "GET /tag/copyright/ HTTP/1.0" 200 37874 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
46.4.132.226 - - [21/Apr/2015:03:52:25 +0200] "GET /tag/copyright/feed/ HTTP/1.0" 200 17376 "http://www.gridshore.nl/tag/copyright/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
162.243.222.48 - - [21/Apr/2015:03:52:31 +0200] "GET /feed/atom/ HTTP/1.1" 200 70930 "-" "Feedbin - 1 subscribers"
183.161.168.153 - - [21/Apr/2015:03:52:32 +0200] "GET /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 200 34606 "-" "Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1)"
183.161.168.153 - - [21/Apr/2015:03:52:34 +0200] "POST /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 302 790 "http://www.gridshore.nl/2014/08/15/creativity-inc-by-ed-catmull/" "Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1; SV1)"
183.161.168.153 - - [21/Apr/2015:03:52:36 +0200] "GET /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 200 34607 "-" "Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1; SV1) ; .NET CLR 2.0.50727 ; .NET
CLR 4.0.30319)"
46.4.132.226 - - [21/Apr/2015:03:52:38 +0200] "GET /tag/openness/ HTTP/1.0" 200 37864 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
66.249.65.32 - - [21/Apr/2015:03:52:47 +0200] "GET /feed HTTP/1.1" 301 451 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.32 - - [21/Apr/2015:03:52:48 +0200] "GET /feed/ HTTP/1.1" 304 243 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
46.4.132.226 - - [21/Apr/2015:03:52:52 +0200] "GET /2008/10/15/time-for-a-new-editor-and-other-vacation-musings/ HTTP/1.0" 200 46014 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://
www.seokicks.de/robot.html)"
46.4.132.226 - - [21/Apr/2015:03:53:06 +0200] "GET /2008/10/19/one-liter-of-guice-during-spring-break/ HTTP/1.0" 200 72731 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/
robot.html)"
46.4.132.226 - - [21/Apr/2015:03:53:21 +0200] "GET /2008/10/19/one-liter-of-guice-during-spring-break/feed/ HTTP/1.0" 200 10136 "http://www.gridshore.nl/2008/10/19/one-liter-of-guice-during-spring-break/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://
www.seokicks.de/robot.html)"
http://serverfault.com/questions/11028/do-you-have-any-useful-awk-and-grep-scripts-for-parsing-apache-logs

14. indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?js=wp-includes%2Fjs%2Fjquery%2Fjquery.js%3Fver%3D1.11.1%2Cwp-includes%2Fjs%2Fjquery%2Fjquery-migrate.min.js%3Fver%3D1.2.1%2Cwp-content%2Fthemes
%2Fspacious%2Fjs%2Fspacious-custom.js%3Fver%3D4.1.1 HTTP/1.1" 200 36442 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/akismet/_inc/form.js?ver=3.1.1 HTTP/1.1" 200 969 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css
%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3 HTTP/1.1" 200 36995 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/jetpack/modules/wpgroho.js?ver=4.1.1 HTTP/1.1" 200 1228 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X
10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/scripts/shBrushJScript.js?ver=3.0.9b HTTP/1.1" 200 2056 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0
(Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/scripts/shCore.js?ver=3.0.9b HTTP/1.1" 200 24190 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0
(Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-includes/js/comment-reply.min.js?ver=4.1.1 HTTP/1.1" 200 1026 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/themes/spacious/js/navigation.js?ver=4.1.1 HTTP/1.1" 200 1233 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X
10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/jetpack/modules/sharedaddy/sharing.js?ver=3.4.3 HTTP/1.1" 200 43124 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/styles/shCore.css?ver=3.0.9b HTTP/1.1" 200 7042 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0
(Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/styles/shThemeDefault.css?ver=3.0.9b HTTP/1.1" 200 3257 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/
5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-admin/admin-ajax.php?action=wordfence_logHuman&hid=57D2A62DA174F5DA747EC194B32CBD32&r=0.9168877548072487 HTTP/1.1" 200 461 "http://www.gridshore.nl/2014/07/26/transform-the-input-
before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/jetpack/_inc/genericons/genericons/svg HTTP/1.1" 404 32544 "http://www.gridshore.nl/wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver
%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:36 +0200] "GET /favicon.ico HTTP/1.1" 200 2829 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like
Gecko) Chrome/42.0.2311.90 Safari/537.36"
46.4.132.226 - - [21/Apr/2015:03:51:37 +0200] "GET /tag/united-states/feed/ HTTP/1.0" 200 15341 "http://www.gridshore.nl/tag/united-states/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
46.4.132.226 - - [21/Apr/2015:03:51:47 +0200] "GET /2008/11/04/threadsafe-applications-part-i/ HTTP/1.0" 200 57529 "http://www.gridshore.nl/2008/11/05/yes-we-canthe-day-after/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
46.4.132.226 - - [21/Apr/2015:03:52:01 +0200] "GET /2008/11/04/threadsafe-applications-part-i/feed/ HTTP/1.0" 200 7159 "http://www.gridshore.nl/2008/11/04/threadsafe-applications-part-i/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/
robot.html)"
54.197.215.119 - - [21/Apr/2015:03:52:10 +0200] "GET /favicon.ico HTTP/1.1" 200 2829 "-" "Pinterest/0.2 (+http://www.pinterest.com/)"
54.197.215.119 - - [21/Apr/2015:03:52:11 +0200] "GET /apple-touch-icon.png HTTP/1.1" 404 32507 "-" "Pinterest/0.2 (+http://www.pinterest.com/)"
46.4.132.226 - - [21/Apr/2015:03:52:12 +0200] "GET /tag/copyright/ HTTP/1.0" 200 37874 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
46.4.132.226 - - [21/Apr/2015:03:52:25 +0200] "GET /tag/copyright/feed/ HTTP/1.0" 200 17376 "http://www.gridshore.nl/tag/copyright/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
162.243.222.48 - - [21/Apr/2015:03:52:31 +0200] "GET /feed/atom/ HTTP/1.1" 200 70930 "-" "Feedbin - 1 subscribers"
183.161.168.153 - - [21/Apr/2015:03:52:32 +0200] "GET /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 200 34606 "-" "Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1)"
183.161.168.153 - - [21/Apr/2015:03:52:34 +0200] "POST /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 302 790 "http://www.gridshore.nl/2014/08/15/creativity-inc-by-ed-catmull/" "Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1; SV1)"
183.161.168.153 - - [21/Apr/2015:03:52:36 +0200] "GET /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 200 34607 "-" "Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1; SV1) ; .NET CLR 2.0.50727 ; .NET
CLR 4.0.30319)"
46.4.132.226 - - [21/Apr/2015:03:52:38 +0200] "GET /tag/openness/ HTTP/1.0" 200 37864 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
66.249.65.32 - - [21/Apr/2015:03:52:47 +0200] "GET /feed HTTP/1.1" 301 451 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.32 - - [21/Apr/2015:03:52:48 +0200] "GET /feed/ HTTP/1.1" 304 243 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
46.4.132.226 - - [21/Apr/2015:03:52:52 +0200] "GET /2008/10/15/time-for-a-new-editor-and-other-vacation-musings/ HTTP/1.0" 200 46014 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://
www.seokicks.de/robot.html)"
46.4.132.226 - - [21/Apr/2015:03:53:06 +0200] "GET /2008/10/19/one-liter-of-guice-during-spring-break/ HTTP/1.0" 200 72731 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/
robot.html)"
46.4.132.226 - - [21/Apr/2015:03:53:21 +0200] "GET /2008/10/19/one-liter-of-guice-during-spring-break/feed/ HTTP/1.0" 200 10136 "http://www.gridshore.nl/2008/10/19/one-liter-of-guice-during-spring-break/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://
www.seokicks.de/robot.html)"
# tail -fn 100 access-log-2014-04-22
http://serverfault.com/questions/11028/do-you-have-any-useful-awk-and-grep-scripts-for-parsing-apache-logs

15. indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?js=wp-includes%2Fjs%2Fjquery%2Fjquery.js%3Fver%3D1.11.1%2Cwp-includes%2Fjs%2Fjquery%2Fjquery-migrate.min.js%3Fver%3D1.2.1%2Cwp-content%2Fthemes
%2Fspacious%2Fjs%2Fspacious-custom.js%3Fver%3D4.1.1 HTTP/1.1" 200 36442 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/akismet/_inc/form.js?ver=3.1.1 HTTP/1.1" 200 969 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css
%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3 HTTP/1.1" 200 36995 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/jetpack/modules/wpgroho.js?ver=4.1.1 HTTP/1.1" 200 1228 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X
10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/scripts/shBrushJScript.js?ver=3.0.9b HTTP/1.1" 200 2056 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0
(Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/scripts/shCore.js?ver=3.0.9b HTTP/1.1" 200 24190 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0
(Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-includes/js/comment-reply.min.js?ver=4.1.1 HTTP/1.1" 200 1026 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/themes/spacious/js/navigation.js?ver=4.1.1 HTTP/1.1" 200 1233 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X
10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/jetpack/modules/sharedaddy/sharing.js?ver=3.4.3 HTTP/1.1" 200 43124 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/styles/shCore.css?ver=3.0.9b HTTP/1.1" 200 7042 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0
(Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/styles/shThemeDefault.css?ver=3.0.9b HTTP/1.1" 200 3257 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/
5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-admin/admin-ajax.php?action=wordfence_logHuman&hid=57D2A62DA174F5DA747EC194B32CBD32&r=0.9168877548072487 HTTP/1.1" 200 461 "http://www.gridshore.nl/2014/07/26/transform-the-input-
before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/jetpack/_inc/genericons/genericons/svg HTTP/1.1" 404 32544 "http://www.gridshore.nl/wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver
%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:36 +0200] "GET /favicon.ico HTTP/1.1" 200 2829 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like
Gecko) Chrome/42.0.2311.90 Safari/537.36"
46.4.132.226 - - [21/Apr/2015:03:51:37 +0200] "GET /tag/united-states/feed/ HTTP/1.0" 200 15341 "http://www.gridshore.nl/tag/united-states/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
46.4.132.226 - - [21/Apr/2015:03:51:47 +0200] "GET /2008/11/04/threadsafe-applications-part-i/ HTTP/1.0" 200 57529 "http://www.gridshore.nl/2008/11/05/yes-we-canthe-day-after/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
46.4.132.226 - - [21/Apr/2015:03:52:01 +0200] "GET /2008/11/04/threadsafe-applications-part-i/feed/ HTTP/1.0" 200 7159 "http://www.gridshore.nl/2008/11/04/threadsafe-applications-part-i/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/
robot.html)"
54.197.215.119 - - [21/Apr/2015:03:52:10 +0200] "GET /favicon.ico HTTP/1.1" 200 2829 "-" "Pinterest/0.2 (+http://www.pinterest.com/)"
54.197.215.119 - - [21/Apr/2015:03:52:11 +0200] "GET /apple-touch-icon.png HTTP/1.1" 404 32507 "-" "Pinterest/0.2 (+http://www.pinterest.com/)"
46.4.132.226 - - [21/Apr/2015:03:52:12 +0200] "GET /tag/copyright/ HTTP/1.0" 200 37874 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
46.4.132.226 - - [21/Apr/2015:03:52:25 +0200] "GET /tag/copyright/feed/ HTTP/1.0" 200 17376 "http://www.gridshore.nl/tag/copyright/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
162.243.222.48 - - [21/Apr/2015:03:52:31 +0200] "GET /feed/atom/ HTTP/1.1" 200 70930 "-" "Feedbin - 1 subscribers"
183.161.168.153 - - [21/Apr/2015:03:52:32 +0200] "GET /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 200 34606 "-" "Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1)"
183.161.168.153 - - [21/Apr/2015:03:52:34 +0200] "POST /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 302 790 "http://www.gridshore.nl/2014/08/15/creativity-inc-by-ed-catmull/" "Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1; SV1)"
183.161.168.153 - - [21/Apr/2015:03:52:36 +0200] "GET /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 200 34607 "-" "Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1; SV1) ; .NET CLR 2.0.50727 ; .NET
CLR 4.0.30319)"
46.4.132.226 - - [21/Apr/2015:03:52:38 +0200] "GET /tag/openness/ HTTP/1.0" 200 37864 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
66.249.65.32 - - [21/Apr/2015:03:52:47 +0200] "GET /feed HTTP/1.1" 301 451 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.32 - - [21/Apr/2015:03:52:48 +0200] "GET /feed/ HTTP/1.1" 304 243 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
46.4.132.226 - - [21/Apr/2015:03:52:52 +0200] "GET /2008/10/15/time-for-a-new-editor-and-other-vacation-musings/ HTTP/1.0" 200 46014 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://
www.seokicks.de/robot.html)"
46.4.132.226 - - [21/Apr/2015:03:53:06 +0200] "GET /2008/10/19/one-liter-of-guice-during-spring-break/ HTTP/1.0" 200 72731 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/
robot.html)"
46.4.132.226 - - [21/Apr/2015:03:53:21 +0200] "GET /2008/10/19/one-liter-of-guice-during-spring-break/feed/ HTTP/1.0" 200 10136 "http://www.gridshore.nl/2008/10/19/one-liter-of-guice-during-spring-break/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://
www.seokicks.de/robot.html)"
# tail -fn 100 access-log-2014-04-22
awk -F'[ "]+' '$7 == "/" { ipcount[$1]++ }
END { for (i in ipcount) {
printf "%15s - %dn", i, ipcount[i] } }' access-log-2015-04-21
http://serverfault.com/questions/11028/do-you-have-any-useful-awk-and-grep-scripts-for-parsing-apache-logs

16. EVERY NIGHT A BATCH USING WEBALIZER

17. GOOGLE ANALYTICS

18. GOOGLE ANALYTICS

19. GOOGLE ANALYTICS

20. WHAT IS REALTIME?

21. THERE IS ALWAYS A DELAY

22. HOW MUCH DELAY CAN
YOU ACCEPT?

23. ARCHITECTURE OF DELAY
access
logs shipper Queue Logstash
elasticsearch
Monitor Send Retrieve
Store
forwarder
logstash
beaver
Redis
Kafka

24. DATA LIFECYCLE

25. DATA LIFECYCLE
Obtain

26. DATA LIFECYCLE
Obtain Transform

27. DATA LIFECYCLE
Obtain Transform Store

28. DATA LIFECYCLE
Obtain Transform Store Use

29. DATA LIFECYCLE
Obtain Transform Store Use
Learn

30. DATA LIFECYCLE: ELK
Obtain Transform Store Use
Learn

31. DATA LIFECYCLE: ELK
Obtain Transform Store Use
Learn
Logstash

32. DATA LIFECYCLE: ELK
Obtain Transform Store Use
Learn
Logstash Logstash

33. DATA LIFECYCLE: ELK
Obtain Transform Store Use
Learn
Logstash Logstash Elasticsearch

34. DATA LIFECYCLE: ELK
Obtain Transform Store Use
Learn
Logstash Logstash Elasticsearch Kibana

35. DATA LIFECYCLE: ELK
Obtain Transform Store Use
Learn
Logstash Logstash Elasticsearch Kibana
YOU

36. INTRODUCTION OF ELK
COMPONENTS

37. INTRODUCTION OF ELK
COMPONENTS

38. LOGSTASH: COMPONENTS
file
syslog
redis log4j
web
socket
twitter grok
mutate
drop
clone
geoipelastic
search
file
graphite
statsd

39. LOGSTASH: COMPONENTS
file
syslog
redis log4j
Input
web
socket
twitter
Filter
Output
grok
mutate
drop
clone
geoipelastic
search
file
graphite
statsd

40. LOGSTASH: COMPONENTS
file syslog redislog4jInput
web
socket
twitter
Filter
Output
grok mutatedropclone geoip
elastic
search
filegraphite statsd

41. ELASTICSEARCH

42. ELASTICSEARCH
cluster

43. ELASTICSEARCH
cluster
Node Node Node

44. ELASTICSEARCH
cluster
Node Node Node
Index
Index
Index
Index
Index
Index

45. ELASTICSEARCH
cluster
Node Node Node
Index
Index
Index
Index
Index
Index
shardshard
shardshard
shardshard
shardshard
shardshard
shardshard

46. ELASTICSEARCH
cluster
Node Node Node
Index
Index
Index
Index
Index
Index
shardshard
shardshard
shardshard
shardshard
shardshard
shardshard
Mapping

47. ELASTICSEARCH
cluster
Node Node Node
Index
Index
Index
Index
Index
Index
shardshard
shardshard
shardshard
shardshard
shardshard
shardshard
Mapping
Search API

48. ELASTICSEARCH
cluster
Node Node Node
Index
Index
Index
Index
Index
Index
shardshard
shardshard
shardshard
shardshard
shardshard
shardshard
Mapping
Search API
Aggregations

49. AGGREGATIONS

50. AGGREGATIONS
27.159.213.10 - - [25/Feb/2015:03:35:57 +0100] "GET /2008/04/29/using-ehcache-and-
verifying-that-it-works-with-jpa-and-springframework/ HTTP/1.1" 200 18720 "-" "Mozilla/4.0
(compatible; MSIE 6.0;Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0;Windows
NT 5.1; SV1) ; .NET CLR 1.0.3705)"

51. AGGREGATIONS
27.159.213.10 - - [25/Feb/2015:03:35:57 +0100] "GET /2008/04/29/using-ehcache-and-
verifying-that-it-works-with-jpa-and-springframework/ HTTP/1.1" 200 18720 "-" "Mozilla/4.0
(compatible; MSIE 6.0;Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0;Windows
NT 5.1; SV1) ; .NET CLR 1.0.3705)"
GET

52. AGGREGATIONS
27.159.213.10 - - [25/Feb/2015:03:35:57 +0100] "GET /2008/04/29/using-ehcache-and-
verifying-that-it-works-with-jpa-and-springframework/ HTTP/1.1" 200 18720 "-" "Mozilla/4.0
(compatible; MSIE 6.0;Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0;Windows
NT 5.1; SV1) ; .NET CLR 1.0.3705)"

53. AGGREGATIONS
78.5.169.90 - - [15/Apr/2015:03:08:17 +0200] "POST /wp-login.php HTTP/1.1" 403 3538
"http://www.gridshore.nl/wp-login.php" "Mozilla/5.0 (Windows NT 6.1;WOW64)
AppleWebKit/537.17 (KHTML like Gecko) Chrome/24.0.1312.56 Safari/537.17"

54. AGGREGATIONS
78.5.169.90 - - [15/Apr/2015:03:08:17 +0200] "POST /wp-login.php HTTP/1.1" 403 3538
"http://www.gridshore.nl/wp-login.php" "Mozilla/5.0 (Windows NT 6.1;WOW64)
AppleWebKit/537.17 (KHTML like Gecko) Chrome/24.0.1312.56 Safari/537.17"
POST

55. AGGREGATIONS
78.5.169.90 - - [15/Apr/2015:03:08:17 +0200] "POST /wp-login.php HTTP/1.1" 403 3538
"http://www.gridshore.nl/wp-login.php" "Mozilla/5.0 (Windows NT 6.1;WOW64)
AppleWebKit/537.17 (KHTML like Gecko) Chrome/24.0.1312.56 Safari/537.17"

56. AGGREGATIONS
175989 133343 2008 2
POSTGET HEAD PUT

57. AGGREGATIONS
Date histogram
Feb Mar Apr
311344 395654 157623

58. AGGREGATIONS
Date histogram
Feb Mar Apr
311344 395654 157623
Cardinality [client ip] 11848 26152 9064

59. GET /gridshore-logs-*/_search?search_type=count
{
"aggs": {
"byDate": {
"date_histogram": {
"field": "@timestamp",
"interval": "month"
},
"aggs": {
"uniqueVisitors": {
"cardinality": {
"field": "clientip"
}
}
}
}
}
}

60. KIBANA
Discover

61. KIBANA
Discover
Visualise

62. KIBANA
Discover
Visualise
Analyse

63. Discover

64. Visualise

65. Analyse

66. WHO IS ABUSING MY BLOG?

67. OBTAINING LOGS
daily rolling file

68. OBTAINING LOGS
daily rolling file
shell script ftp

69. OBTAINING LOGS
daily rolling file
shell script ftp
logstash

70. OBTAINING LOGS
daily rolling file
shell script ftp
logstash
elasticsearch

71. 1. input {
2. file {
3. path => "/access-log-*"
4. type => "apache"
5. start_position => "beginning"
6. }
7. }
OBTAIN

72. 1. input {
2. file {
3. path => "/access-log-*"
4. type => "apache"
5. start_position => "beginning"
6. }
7. }
OBTAIN
files to import

73. 1. input {
2. file {
3. path => "/access-log-*"
4. type => "apache"
5. start_position => "beginning"
6. }
7. }
OBTAIN
used for filtering

74. 1. input {
2. file {
3. path => "/access-log-*"
4. type => "apache"
5. start_position => "beginning"
6. }
7. }
OBTAIN
start reading from

75. indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?js=wp-includes%2Fjs%2Fjquery%2Fjquery.js%3Fver%3D1.11.1%2Cwp-includes%2Fjs%2Fjquery%2Fjquery-migrate.min.js%3Fver%3D1.2.1%2Cwp-content%2Fthemes
%2Fspacious%2Fjs%2Fspacious-custom.js%3Fver%3D4.1.1 HTTP/1.1" 200 36442 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/akismet/_inc/form.js?ver=3.1.1 HTTP/1.1" 200 969 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css
%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3 HTTP/1.1" 200 36995 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/jetpack/modules/wpgroho.js?ver=4.1.1 HTTP/1.1" 200 1228 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X
10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/scripts/shBrushJScript.js?ver=3.0.9b HTTP/1.1" 200 2056 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0
(Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/scripts/shCore.js?ver=3.0.9b HTTP/1.1" 200 24190 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0
(Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-includes/js/comment-reply.min.js?ver=4.1.1 HTTP/1.1" 200 1026 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/themes/spacious/js/navigation.js?ver=4.1.1 HTTP/1.1" 200 1233 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X
10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/jetpack/modules/sharedaddy/sharing.js?ver=3.4.3 HTTP/1.1" 200 43124 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/styles/shCore.css?ver=3.0.9b HTTP/1.1" 200 7042 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0
(Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/styles/shThemeDefault.css?ver=3.0.9b HTTP/1.1" 200 3257 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/
5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-admin/admin-ajax.php?action=wordfence_logHuman&hid=57D2A62DA174F5DA747EC194B32CBD32&r=0.9168877548072487 HTTP/1.1" 200 461 "http://www.gridshore.nl/2014/07/26/transform-the-input-
before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/jetpack/_inc/genericons/genericons/svg HTTP/1.1" 404 32544 "http://www.gridshore.nl/wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver
%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:36 +0200] "GET /favicon.ico HTTP/1.1" 200 2829 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like
Gecko) Chrome/42.0.2311.90 Safari/537.36"
46.4.132.226 - - [21/Apr/2015:03:51:37 +0200] "GET /tag/united-states/feed/ HTTP/1.0" 200 15341 "http://www.gridshore.nl/tag/united-states/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
46.4.132.226 - - [21/Apr/2015:03:51:47 +0200] "GET /2008/11/04/threadsafe-applications-part-i/ HTTP/1.0" 200 57529 "http://www.gridshore.nl/2008/11/05/yes-we-canthe-day-after/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
46.4.132.226 - - [21/Apr/2015:03:52:01 +0200] "GET /2008/11/04/threadsafe-applications-part-i/feed/ HTTP/1.0" 200 7159 "http://www.gridshore.nl/2008/11/04/threadsafe-applications-part-i/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/
robot.html)"
54.197.215.119 - - [21/Apr/2015:03:52:10 +0200] "GET /favicon.ico HTTP/1.1" 200 2829 "-" "Pinterest/0.2 (+http://www.pinterest.com/)"
54.197.215.119 - - [21/Apr/2015:03:52:11 +0200] "GET /apple-touch-icon.png HTTP/1.1" 404 32507 "-" "Pinterest/0.2 (+http://www.pinterest.com/)"
46.4.132.226 - - [21/Apr/2015:03:52:12 +0200] "GET /tag/copyright/ HTTP/1.0" 200 37874 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
46.4.132.226 - - [21/Apr/2015:03:52:25 +0200] "GET /tag/copyright/feed/ HTTP/1.0" 200 17376 "http://www.gridshore.nl/tag/copyright/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
162.243.222.48 - - [21/Apr/2015:03:52:31 +0200] "GET /feed/atom/ HTTP/1.1" 200 70930 "-" "Feedbin - 1 subscribers"
183.161.168.153 - - [21/Apr/2015:03:52:32 +0200] "GET /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 200 34606 "-" "Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1)"
183.161.168.153 - - [21/Apr/2015:03:52:34 +0200] "POST /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 302 790 "http://www.gridshore.nl/2014/08/15/creativity-inc-by-ed-catmull/" "Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1; SV1)"
183.161.168.153 - - [21/Apr/2015:03:52:36 +0200] "GET /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 200 34607 "-" "Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1; SV1) ; .NET CLR 2.0.50727 ; .NET
CLR 4.0.30319)"
46.4.132.226 - - [21/Apr/2015:03:52:38 +0200] "GET /tag/openness/ HTTP/1.0" 200 37864 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
66.249.65.32 - - [21/Apr/2015:03:52:47 +0200] "GET /feed HTTP/1.1" 301 451 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.32 - - [21/Apr/2015:03:52:48 +0200] "GET /feed/ HTTP/1.1" 304 243 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
46.4.132.226 - - [21/Apr/2015:03:52:52 +0200] "GET /2008/10/15/time-for-a-new-editor-and-other-vacation-musings/ HTTP/1.0" 200 46014 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://
www.seokicks.de/robot.html)"
46.4.132.226 - - [21/Apr/2015:03:53:06 +0200] "GET /2008/10/19/one-liter-of-guice-during-spring-break/ HTTP/1.0" 200 72731 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/
robot.html)"
46.4.132.226 - - [21/Apr/2015:03:53:21 +0200] "GET /2008/10/19/one-liter-of-guice-during-spring-break/feed/ HTTP/1.0" 200 10136 "http://www.gridshore.nl/2008/10/19/one-liter-of-guice-during-spring-break/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://
www.seokicks.de/robot.html)"

76. indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?js=wp-includes%2Fjs%2Fjquery%2Fjquery.js%3Fver%3D1.11.1%2Cwp-includes%2Fjs%2Fjquery%2Fjquery-migrate.min.js%3Fver%3D1.2.1%2Cwp-content%2Fthemes
%2Fspacious%2Fjs%2Fspacious-custom.js%3Fver%3D4.1.1 HTTP/1.1" 200 36442 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/akismet/_inc/form.js?ver=3.1.1 HTTP/1.1" 200 969 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css
%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3 HTTP/1.1" 200 36995 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/jetpack/modules/wpgroho.js?ver=4.1.1 HTTP/1.1" 200 1228 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X
10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/scripts/shBrushJScript.js?ver=3.0.9b HTTP/1.1" 200 2056 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0
(Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/scripts/shCore.js?ver=3.0.9b HTTP/1.1" 200 24190 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0
(Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-includes/js/comment-reply.min.js?ver=4.1.1 HTTP/1.1" 200 1026 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/themes/spacious/js/navigation.js?ver=4.1.1 HTTP/1.1" 200 1233 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X
10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/jetpack/modules/sharedaddy/sharing.js?ver=3.4.3 HTTP/1.1" 200 43124 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/styles/shCore.css?ver=3.0.9b HTTP/1.1" 200 7042 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0
(Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/styles/shThemeDefault.css?ver=3.0.9b HTTP/1.1" 200 3257 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/
5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-admin/admin-ajax.php?action=wordfence_logHuman&hid=57D2A62DA174F5DA747EC194B32CBD32&r=0.9168877548072487 HTTP/1.1" 200 461 "http://www.gridshore.nl/2014/07/26/transform-the-input-
before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/jetpack/_inc/genericons/genericons/svg HTTP/1.1" 404 32544 "http://www.gridshore.nl/wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver
%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:36 +0200] "GET /favicon.ico HTTP/1.1" 200 2829 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like
Gecko) Chrome/42.0.2311.90 Safari/537.36"
46.4.132.226 - - [21/Apr/2015:03:51:37 +0200] "GET /tag/united-states/feed/ HTTP/1.0" 200 15341 "http://www.gridshore.nl/tag/united-states/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
46.4.132.226 - - [21/Apr/2015:03:51:47 +0200] "GET /2008/11/04/threadsafe-applications-part-i/ HTTP/1.0" 200 57529 "http://www.gridshore.nl/2008/11/05/yes-we-canthe-day-after/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
46.4.132.226 - - [21/Apr/2015:03:52:01 +0200] "GET /2008/11/04/threadsafe-applications-part-i/feed/ HTTP/1.0" 200 7159 "http://www.gridshore.nl/2008/11/04/threadsafe-applications-part-i/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/
robot.html)"
54.197.215.119 - - [21/Apr/2015:03:52:10 +0200] "GET /favicon.ico HTTP/1.1" 200 2829 "-" "Pinterest/0.2 (+http://www.pinterest.com/)"
54.197.215.119 - - [21/Apr/2015:03:52:11 +0200] "GET /apple-touch-icon.png HTTP/1.1" 404 32507 "-" "Pinterest/0.2 (+http://www.pinterest.com/)"
46.4.132.226 - - [21/Apr/2015:03:52:12 +0200] "GET /tag/copyright/ HTTP/1.0" 200 37874 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
46.4.132.226 - - [21/Apr/2015:03:52:25 +0200] "GET /tag/copyright/feed/ HTTP/1.0" 200 17376 "http://www.gridshore.nl/tag/copyright/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
162.243.222.48 - - [21/Apr/2015:03:52:31 +0200] "GET /feed/atom/ HTTP/1.1" 200 70930 "-" "Feedbin - 1 subscribers"
183.161.168.153 - - [21/Apr/2015:03:52:32 +0200] "GET /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 200 34606 "-" "Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1)"
183.161.168.153 - - [21/Apr/2015:03:52:34 +0200] "POST /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 302 790 "http://www.gridshore.nl/2014/08/15/creativity-inc-by-ed-catmull/" "Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1; SV1)"
183.161.168.153 - - [21/Apr/2015:03:52:36 +0200] "GET /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 200 34607 "-" "Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1; SV1) ; .NET CLR 2.0.50727 ; .NET
CLR 4.0.30319)"
46.4.132.226 - - [21/Apr/2015:03:52:38 +0200] "GET /tag/openness/ HTTP/1.0" 200 37864 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
66.249.65.32 - - [21/Apr/2015:03:52:47 +0200] "GET /feed HTTP/1.1" 301 451 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.32 - - [21/Apr/2015:03:52:48 +0200] "GET /feed/ HTTP/1.1" 304 243 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
46.4.132.226 - - [21/Apr/2015:03:52:52 +0200] "GET /2008/10/15/time-for-a-new-editor-and-other-vacation-musings/ HTTP/1.0" 200 46014 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://
www.seokicks.de/robot.html)"
46.4.132.226 - - [21/Apr/2015:03:53:06 +0200] "GET /2008/10/19/one-liter-of-guice-during-spring-break/ HTTP/1.0" 200 72731 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/
robot.html)"
46.4.132.226 - - [21/Apr/2015:03:53:21 +0200] "GET /2008/10/19/one-liter-of-guice-during-spring-break/feed/ HTTP/1.0" 200 10136 "http://www.gridshore.nl/2008/10/19/one-liter-of-guice-during-spring-break/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://
www.seokicks.de/robot.html)"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/
gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver
%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons
%2Fgenericons.css%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss
%2Fjetpack.css%3Fver%3D3.4.3 HTTP/1.1" 200 36995 "http://www.gridshore.nl/
2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0
(Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/42.0.2311.90 Safari/537.36"

78. 1. filter {
2. grok {
3. match => { "message" => "%{COMBINEDAPACHELOG}" }
4. remove_field => ["message"]
5. }
6. grok {
7. match => { "request" => "%{URIPATH:request_noparam}"}
8. }
9. geoip {
10. source => "clientip"
11. }
12. useragent {
13. source => "agent"
14. target => "useragent"
15. remove_field => ["agent"]
16. }
17. date {
18. match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
19. }
20.}
TRANSFORM

79. 1. filter {
2. grok {
3. match => { "message" => "%{COMBINEDAPACHELOG}" }
4. remove_field => ["message"]
5. }
6. grok {
7. match => { "request" => "%{URIPATH:request_noparam}"}
8. }
9. geoip {
10. source => "clientip"
11. }
12. useragent {
13. source => "agent"
14. target => "useragent"
15. remove_field => ["agent"]
16. }
17. date {
18. match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
19. }
20.}
remove parsed
message
TRANSFORM

80. 1. filter {
2. grok {
3. match => { "message" => "%{COMBINEDAPACHELOG}" }
4. remove_field => ["message"]
5. }
6. grok {
7. match => { "request" => "%{URIPATH:request_noparam}"}
8. }
9. geoip {
10. source => "clientip"
11. }
12. useragent {
13. source => "agent"
14. target => "useragent"
15. remove_field => ["agent"]
16. }
17. date {
18. match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
19. }
20.}
TRANSFORM
extra parse of request

81. 1. filter {
2. grok {
3. match => { "message" => "%{COMBINEDAPACHELOG}" }
4. remove_field => ["message"]
5. }
6. grok {
7. match => { "request" => "%{URIPATH:request_noparam}"}
8. }
9. geoip {
10. source => "clientip"
11. }
12. useragent {
13. source => "agent"
14. target => "useragent"
15. remove_field => ["agent"]
16. }
17. date {
18. match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
19. }
20.}
TRANSFORM
request => /wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes
%2Fspacious-child%2Fstyle.css%3Fver%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack
%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css%3Fver%3D3.1%2Cwp-content
%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3
request_noparam => /wp-content/plugins/scripts-gzip/gzip.php

82. 1. filter {
2. grok {
3. match => { "message" => "%{COMBINEDAPACHELOG}" }
4. remove_field => ["message"]
5. }
6. grok {
7. match => { "request" => "%{URIPATH:request_noparam}"}
8. }
9. geoip {
10. source => "clientip"
11. }
12. useragent {
13. source => "agent"
14. target => "useragent"
15. remove_field => ["agent"]
16. }
17. date {
18. match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
19. }
20.}
TRANSFORM
add geo information

83. 1. filter {
2. grok {
3. match => { "message" => "%{COMBINEDAPACHELOG}" }
4. remove_field => ["message"]
5. }
6. grok {
7. match => { "request" => "%{URIPATH:request_noparam}"}
8. }
9. geoip {
10. source => "clientip"
11. }
12. useragent {
13. source => "agent"
14. target => "useragent"
15. remove_field => ["agent"]
16. }
17. date {
18. match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
19. }
20.}
TRANSFORM
parse useragent
fields

84. 1. filter {
2. grok {
3. match => { "message" => "%{COMBINEDAPACHELOG}" }
4. remove_field => ["message"]
5. }
6. grok {
7. match => { "request" => "%{URIPATH:request_noparam}"}
8. }
9. geoip {
10. source => "clientip"
11. }
12. useragent {
13. source => "agent"
14. target => "useragent"
15. remove_field => ["agent"]
16. }
17. date {
18. match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
19. }
20.}
TRANSFORM
agent => Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36
(KHTML, like Gecko) Safari/537.36
useragent => {"name": "Safari", "os":“Mac OS X 10.10.2”, "os_name":“Mac OS X”,
"device": "Other", "major": "537", "minor": "36"
}

85. 1. filter {
2. grok {
3. match => { "message" => "%{COMBINEDAPACHELOG}" }
4. remove_field => ["message"]
5. }
6. grok {
7. match => { "request" => "%{URIPATH:request_noparam}"}
8. }
9. geoip {
10. source => "clientip"
11. }
12. useragent {
13. source => "agent"
14. target => "useragent"
15. remove_field => ["agent"]
16. }
17. date {
18. match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
19. }
20.}
TRANSFORM
take timestamp
from log

86. STORE
1. output {
2. if "_grokparsefailure" not in [tags] {
3. elasticsearch {
4. protocol => "transport"
5. host => "localhost:9300"
6. cluster => "jc-play"
7. index => "gridshore-logs-%{+YYYY.MM}"
8. manage_template => false
9. template_name => "gridshore-logs"
10. }
11. }
12.}

87. STORE
1. output {
2. if "_grokparsefailure" not in [tags] {
3. elasticsearch {
4. protocol => "transport"
5. host => "localhost:9300"
6. cluster => "jc-play"
7. index => "gridshore-logs-%{+YYYY.MM}"
8. manage_template => false
9. template_name => "gridshore-logs"
10. }
11. }
12.}
in case of an error

88. STORE
1. output {
2. if "_grokparsefailure" not in [tags] {
3. elasticsearch {
4. protocol => "transport"
5. host => "localhost:9300"
6. cluster => "jc-play"
7. index => "gridshore-logs-%{+YYYY.MM}"
8. manage_template => false
9. template_name => "gridshore-logs"
10. }
11. }
12.}
use faster binary
protocol

89. STORE
1. output {
2. if "_grokparsefailure" not in [tags] {
3. elasticsearch {
4. protocol => "transport"
5. host => "localhost:9300"
6. cluster => "jc-play"
7. index => "gridshore-logs-%{+YYYY.MM}"
8. manage_template => false
9. template_name => "gridshore-logs"
10. }
11. }
12.}
format of index to create:
gridshore-logs-2015.02

90. STORE
1. output {
2. if "_grokparsefailure" not in [tags] {
3. elasticsearch {
4. protocol => "transport"
5. host => "localhost:9300"
6. cluster => "jc-play"
7. index => "gridshore-logs-%{+YYYY.MM}"
8. manage_template => false
9. template_name => "gridshore-logs"
10. }
11. }
12.}
provide our own
index template

91. DEMO

92. Integrated
Disease
Management
Control
LESSONS LEARNED

93. DATA ENHANCEMENT

94. PROBLEM
WITH DATES

95. PROBLEM
WITH DATES

96. PROBLEM
WITH DATES

97. PROBLEM
WITH DATES

98. WHAT CANNOT BE DONE

99. THINGS ABOUT AGE

100. THINGS ABOUT AGE

101. GOODTO KNOW

102. GETTING BIG

103. SMAP - Soil Moisture Active Passive
http://smap.jpl.nasa.gov/mission/why-it-matters/
Monitor
Drought
Predict
Floods
Assist
Crop
Productivity
Weather
Forecasting

105. VERIZON
https://speakerdeck.com/bhaskarvk/elastic-on-15-500-billion-documents-and-counting
“We offer technology products and solutions that
transform the way our customers connect, collaborate
and innovate”

106. VERIZON
https://speakerdeck.com/bhaskarvk/elastic-on-15-500-billion-documents-and-counting
“We offer technology products and solutions that
transform the way our customers connect, collaborate
and innovate”
Store massive logging data
Store in high rate
Query in acceptable rate

107. VERIZON
128 Nodes
8 cores - 64 Gb RAM - 6 x 1TB disk
10+ Bilion documents a day
Over 500 Billion documents total

108. SAVINGYOUR DASHBOARDS

110. WHAT ABOUT SECURITY
Elastic shield

111. FUTURE DIRECTIONS

112. LOGSTASH

113. LOGSTASH
• API for pipeline

114. LOGSTASH
• API for pipeline
• Internal / persistent queues

115. LOGSTASH
• API for pipeline
• Internal / persistent queues
• Clustered logstash

116. ELASTICSEARCH

117. ELASTICSEARCH
• Better error responses

118. ELASTICSEARCH
• Better error responses
• Reindex API

119. ELASTICSEARCH
• Better error responses
• Reindex API
• Changes API

120. KIBANA

121. KIBANA
• Formatting output: numbers, currency, urls, video

122. KIBANA
• Formatting output: numbers, currency, urls, video
• Edit and save or pin filters

123. KIBANA
• Formatting output: numbers, currency, urls, video
• Edit and save or pin filters
• Choose your own colours in charts

124. KIBANA
• Formatting output: numbers, currency, urls, video
• Edit and save or pin filters
• Choose your own colours in charts
• Create API for custom plugins

125. SUMMARISE

126. SUMMARISE
• Real time data analysis

127. SUMMARISE
• Real time data analysis
• Obtain and transform data using logstash

128. SUMMARISE
• Real time data analysis
• Obtain and transform data using logstash
• Index data in elasticsearch

129. SUMMARISE
• Real time data analysis
• Obtain and transform data using logstash
• Index data in elasticsearch
• Show data using Kibana

130. SUMMARISE
• Real time data analysis
• Obtain and transform data using logstash
• Index data in elasticsearch
• Show data using Kibana
• What Kibana does well and what not

131. MORE INFORMATION
@jettroCoenradie
jettro.coenradie@luminis.eu
http://amsterdam.luminis.eu/news/
https://www.elastic.co/products