about Real-time Complex Event Processing Technique CEP and Open Source Implementation Esper

1. Complex Event Processing
with Esper
Real-time Event Stream & Complex Event Processing
tedwon

2. 1. Church bells ringing...
2. Appearance of a man in a tuxedo with a woman
in a flowing white gown...
3. Flower flying through the air.
Concept of CEP

3. Concept of CEP
● From these events CEP may infer a wedding.
● CEP as a technique helps discover complex events as a pattern by analyzing
and correlating other events
● http://en.wikipedia.org/wiki/Complex_event_processing#Conceptual_description

4. ● Algorithmic Stock-Trading
● Real-time Stock Tick monitoring
● Detection of credit-card fraud
● Real-time ETL processing
● Business activity monitoring
● Real-time Mobile Targeting Ad & Recommendation
CEP Use Cases

5. CEP is
● based EDA(Event-driven architecture)
● not Request & Response style system
● Detection & Reaction style system
● not Pre-save and Post-process paradigm
● Pre-process and Post-save paradigm
● Loosely-coupled system
● Asynchronous style processing paradigm
● Intelligence system

6. Event Driven Architecture
● Software architecture pattern promoting the production, detection,
consumption of, and reaction to events.
eye,ear...sensory organs think,decision hand,foot...reaction

7. EDA Event Process
1. SEP - Simple Event Processing
a. JMS(Java Message Service)
b. ESB(Enterprise Service Bus)
2. ESP - Event Stream Processing
3. CEP - Complex Event Processing

8. ESP vs. CEP
● ESP
○ Monitor streams of event data, analyse those events, and act upon opportunities
○ Filtering, Aggregation, Join
○ Average of Google stock over the last (moving) 30min
● CEP
○ Detecting patterns among events
○ If this Google stock increased more than 5% two times followed by Apple stock
decreased more than 10% then…

9. CEP Open Source Projects
● JBoss Drools Fusion
○ Rule Script DSL
○ Java
● EsperTech Esper
○ SQL-like Script DSL
○ Java

10. EsperTech Esper

11. Open Source Esper
● Provide ESP/CEP Engine
● Provide SQL-like EPL(Event Processing Language)
● Well-documented reference
● Lightweight & Embeddable

12. Esper Architecture

13. Esper Architecture

14. Esper Architecture
● Real-time ETL application
ETL EPL
(Extract, Transform, Load)
Hadoop FileSystem API ImplThrift Server Impl

15. Esper with Runtime
● Esper + Java EE Platform
● Java Standalone Instance
● Esper + OSGi Server

16. Esper EPL Features
● Event filtering
● Sliding data windows
● Aggregation
● Pattern matching
● Joins and Outer Joins
● Subquery
● Reference historical data

17. Esper EPL Example
Order Top10 by item over last 30 min
OrderEvent.win:time(30 min)from
group by itemId
select itemId, count(*) as cnt
limit 10

18. Esper EPL Example
Continuous query
OrderEvent.win:time(30 min)from
group by itemId
select itemId, count(*) as cnt
limit 10

19. Esper EPL Features

20. MySQL native query
EPL Join
06 EPL과 Adapter 개발
where users visit in Gangnam district
over last one hour
“Place Name!!”
from LR.std:unique(assetId).win:time(1 hour) as lr,
sql:db[select zone_name from ZoneName
where zone=${lr.zone}]
select assetId, zone_name
where zone in (‘gangnam code’)

21. Esper Pattern EPL Example
select Part.zone from pattern [
every Part=CountZone(cnt in (1, 2)) ->
( not CountZone(zone=Part.zone, cnt in (0, 3))
and timer:interval(10 sec) )]

22. Quick Start Esper

23. Esper Quick Start Project

24. Esper Quick Start Project

25. Thank you

26. References
1. https://en.wikipedia.org/wiki/Event-driven_architecture
2. http://en.wikipedia.org/wiki/Complex_event_processing
3. http://www.espertech.com/esper/quickstart.php
4. https://github.com/espertechinc/esper
5. http://www.hawkular.org/blog/2017/01/13/events-aggregation-extension.html
6. https://github.com/tedwon/cep-esper-quick-start
7. https://www.slideshare.net/matthewmccullough/complex-event-processing-with-esper