Real-time Event Stream & Complex Event Processing CEP Overview CEP Usage Esper Overview Esper Example

1. Complex Event Processing
with Esper
Real-time Event Stream & Complex Event Processing
Ted Won
2011. 11. 12.
http://tedwon.com

3. Complex Event Processing

5. Nanosecond
● CEP 세계의 측정 단위는 nanosecond
● A nanosecond (ns) is one billionth of a second

6. Nanosecond
1 second is...
1,000 ms
1,000,000 us
1,000,000,000 ns

7. Concept of CEP
● 교회에서...
● 벨이 울린다.
● 턱시도를 입은 남자와 꽃을 든 여자가 함께 걷는다.
● 두 사람 위로 꽃 잎이 날리고 푹죽이 터진다.

8. Concept of CEP
CEP 기술은 이러한 Complex한 Event를 참조하여
결혼식이라는 것을 알아 차린다.

9. Imagination Beyond Reality
● Imaginary CEP Use Cases

10. Imagination Beyond Reality
● E-mart에서...
● 고객들이 카트를 끌고 매장을 돌아 다닌다.
● 카트의 이동 경로와 패턴을 알 수 없을까?
● 계산대에서 대기하는 고객들 상태 파악 후 실시간 조치
● 카트 + 멤버십 카드라면 카트에 달린 모니터를 통해
개인화된 실시간 추천/쿠폰 발행
● 카트에 물건을 담으면 자동으로 계산

11. Imagination Beyond Reality
● 전국의 이마트 매장이 130개...
● 엄청난 양의 데이터를 어찌 실시간으로 처리한단 말인가??
● Starbucks 매장에서...
● 파주 프리미엄 아웃렛 매장에서...
● 교통 관리에서
실시간으로 교통 상태/패턴을 인지하고 적절한 조치

12. Imagination Beyond Reality
● Algorithmic Stock-Trading
● Applying CEP to the RHQ alert feature
as a Operational Intelligence (OI) solution

13. What is CEP?

14. What is CEP
● Complex Event Processing
● CEP는 Event 중심의 실시간 대용량 데이터 처리 기술
● CEP는 실시간 대용량 Event 처리 기술
● CEP는 EDA(Event-driven architecture) 기반 시스템
● Mutliple event stream에서 발생하는 패턴 감지 기술

15. What is CEP
● Identifying the most meaningful events from event cloud
● Real-time computation and pattern detection over event streams
● Response가 아닌 Reaction을 하는 시스템
● http://en.wikipedia.org/wiki/Complex_Event_Processing

16. What is CEP
● Real-time Processing
● Intelligence System
● Real-time Business Intelligence
● Event Stream Intelligence

17. What is CEP
● Extreme Transaction Processing
● Publish/Subscribe Model
● Loosley-coupled
● Asynchronous Style
Source: Event Processing in Action

18. Open Source CEP
● EsperTech Esper (Stream-oriented)
● JBoss Drools Fusion (Rule-oriented)

19. CEP Language Style
● Strem-oriented
○ EsperTech Esper
○ MS StreamInsight
● Rule-oriented
○ JBoss Drools Fusion
○ TIBCO TIBCO BusinessEvents

20. What is EDA?

21. What is EDA
● Event Driven Architecture
● EDA는 event의 production, detection, consumption과
reaction 하기위한 구조를 정의한 software architecture
Source: Event Processing in Action

22. What is EDA
● More Loosely-coupled than SOA
● Event 표준 스펙 논의가 OASIS에서
Common Base Events라는 이름으로 진행 중
● Response가 아닌 Reaction을 하는 시스템
● 선처리 후저장 패러다임
● http://en.wikipedia.org/wiki/Event-driven_architecture

23. Classical Request-response Architecture
● Synchronous interactions
● Request에 대한 Response를 하는 구조
● 선저장 후처리 패러다임

24. Event Processing Styles
● SEP - Simple Event Processing
○ Single event based
○ 상태가 없는 Event를 처리하는 케이스
○ JMS(Java Message Service)가 대표적인 예
○ ESB(Enterprise Service Bus)
● ESP - Event Stream Processing
● CEP - Complex Event Processing

25. Event Processing Application
Source: Event Processing in Action

26. ESP vs. CEP
● ESP - Event Stream Processing
○ Monitor streams of event data, analyse those events, and act
upon opportunities
○ Volume weighted average of Google stock over the last
(moving) 30min
● CEP - Complex Event Processing
○ Detecting patterns among events
○ If this Google VWAP increased more than 5% two times
followed by Yahoo! VWAP decreased more than 10% then…

27. SOA vs. EDA
● SOA
○ RPC calls
○ Point-to-Point communication
○ Request and reply
○ Vertical interaction
● EDA
○ Message based
○ Less coupled than SOA
○ Publish and subscribe
○ Horizontal communication between tiers in process chain

28. Point-To-Point

29. Publish-and-Subscribe

30. What is an Event?

31. What is an Event
● Event란 무엇인가?
● 무엇을 Event로 정의 할 수 있는가?

32. What is an Event
● Something that happens
● 실제로 발생한 사건, 일, 메세지
● 상태의 변경

33. What is an Event
● An event is an immutable record of a past occurrence
of an action or state change.
● Event properties capture the useful information for an event.

34. What is an Event
Event란 어떤 것이 변화한 것이다!

35. What is an Event
● 변화한다는 것은 어떠한 상태를 가지고 있다는 의미
● 상태를 가지고 변화하는 모든 것이 Event로 정의될 수 있다.
● 이 세상의 모든 것은 변화한다.
● 변화량을 정량적으로 측정가능하도록 정의해야함
● Event를 추상화하여 표현한 것을 Event type이라함

36. What is an Event
● 상태가 없는 Event를 처리하는 경우도 존재
● SEP(Simple Event Processing)의 경우 존재
● JMS가 하나의 예

37. Event Examples
● Stock tick
● 비밀번호 변경
● 10초마다 실내 온도 측정
● 서비스 응답 시간
● 고객이 들고 다가오는 Starbucks membership cards with RFID
● 고객이 끌고 다니는 E-mart carts with RFID

38. Events vs. Messages
● Event는 때로는 Message로 표현됨
● Message는 단순히 Event의 다른 표현이라는 측면도 존재
● Event의 상태라는 속성이 Message에서는 어떤 내용(contents)이
라는 차이
● EDA에 속한 개념으로써 SEP(Simple Event Processing)의 한 종류
인 JMS와 같은 기술에서 사용하는 전문 용어라는 측면

39. What is Event Type?

40. What is Event Type
● Event Representation Schema
● 개념적인 Event를 실제 프로그래밍적으로 표현하는 방법
● Event의 개념적 표현을 Event Schema라 할 수 있는데
● Event Schema의 표현을 Event Type 으로 한다.
● DBMS의 테이블 스키마와 유사한 개념
● 대표적인 Data Format을 이용하여 정의 할 수 있다.

41. Event Type Data Format
● POJO
● Map
● XML
● JSON
● CSV

42. What is Event Stream?

43. What is Event Stream
● 시간의 순서대로 연속되는 Event
● 시작과 끝이 없는 Event의 연속된 흐름

44. What is Real-time?

45. What is Real-time
Real-time system은 다음의 조건들을 충족해야 합니다
● Low latency
● Regularities in Response time
● Providing predictable performance

46. CEP Usages
● Algorithmic Stock-Trading
● Operational Intelligence(OI) solutions
● Real-time analytics
● Predictive analytics
● Event-driven business process management
● BAM - Business Activity Monitoring
● BI - Business intelligence
● BRMSs - Business rule management systems
● Network and System Management
● MOM- Message-oriented middleware
● Stream computing
● ETL(Extract, Transform, Load)

47. CEP Products

48. Esper

49. EsperTech Esper
● Event Stream and Complex Event Processing platform
● Convergence of ESP and CEP
● Open Source ESP/CEP Engine
● Event Processing Language (EPL)
● Java and .Net

50. EsperTech Esper
● High Throughput, Low Latency
● Highly Available (EsperHA)
● Rich Multi-Window GUI (Enterprise Edition)
● Real-time Displays (Enterprise Edition)
● Enterprise *ilities (Enterprise Edition)

51. EsperTech Esper
● Usability
● Maintainability
● Scalability
● Availability and Reliability
● Extensibility
● Security
● Portability

52. EsperTech Esper
● Technical Support (Enterprise Edition)
● Lightweight & Embeddable
● No Runtime Support
● GNU GPL(General Public License) v2

53. Runtime: Esper + Java EE platform

54. Runtime: Esper + OSGi

55. Esper Documentation
● Esper Documentation Home
○ http://goo.gl/isNb6
● Esper 4.4.0 Reference documentation
○ http://goo.gl/5TXel
● EsperIO 4.4.0 Reference documentation
○ http://goo.gl/76uOe

56. Esper Architecture

57. Esper EDAS

58. CEP Architecture

59. CEP Architecture

60. CEP Architecture

61. Sample CEP Architecture

62. Real-time Log Aggregator & CEP

63. Input / Output Adapters
● Binary File Reader Input
● Binary File Writer Output
● CSV File Reader Input
● CSV File Writer Output
● CSV Socket Reader Input
● CSV Socket Writer Output
● Email (IMAP) Reader Input
● Email Sender
● HA Heartbeat Input
● IP Packet Capture /HTTP
● IP Packet Capture /IRC

64. Input / Output Adapters
● IP Packet Capture /POP3
● IP Packet Capture /SMTP
● IRC Output adapter
● Jabber Output Adapter
● JDBC
● JMS Publish/Subscribe*
● Log Output
● Microsoft Excel RTD*
● Multicast UDP Input and Output*
● Quartz Enterprise Scheduler Input*

65. Input / Output Adapters
● Regular Expression File Reader Input
● Regular Expression Socket Reader Input
● RSS
● Twitter
● Unicast UDP Input Adapter
● XML File Writer Output
● XML over HTTP Writer Output
● Hadoop HDFS Input Adapter
● Hadoop HDFS Output Adapter

66. Esper EPL

67. Event Processing Language
select symbol, symbolDesc from OrderEvent.win:time(30 min)

68. Event Processing Language
● No standard
● SQL-like language
● Data windows
● Pattern matching
● Continuous query

69. Event Processing Language

70. Event Processing Model

71. Event Processing Model

72. Event Processing Model

73. Event Processing Model

74. Event Processing Language
● Event filtering
● Sliding windows and aggregation
● Grouped windows and output rate limiting
● Joins and Outer Joins
● Historical or reference data
● Subquery

75. Event Example: The RFID Domain
● RFID: Radio Frequency Identification
● LocationReport
○ asset id - a unique identifier of the tagged asset
○ x - the x location value
○ y - the y location value
○ zone - derived from x and y
● Use cases
○ “When a given group of assets are not moving together from
zone to zone, then...”
○ “When a given asset stays too long in the same zone, then...”

76. EPL Statement Sample
● If a given set of assets are not moving together from zone to zone,
● then alert
insert into CountZone
select zone, count(*) as cnt
from LocationReport.std:unique('assetId')
where assetId in (1, 2, 3)
group by zone

77. EPL Statement Sample
select Part.zone from pattern [
every Part=CountZone(cnt in (1, 2)) ->
( timer:interval(1 min) and not
CountZone(zone=Part.zone, cnt in (0, 3))
)
]

78. Listener and Engine
// Get engine instance and register statement
EPServiceProvider engine = EPServiceProviderManager.getDefaultProvider();
EPStatement statement = engine.getEPAdministrator().createEQL("...");
// Attach a listener
statement.addListener(new UpdateListener() {
public void update(EventBean[] newEvents, EventBean[] oldEvents) {
// Handle complex event
...
}
});

79. Sending events
// Get the same engine instance
EPServiceProvider engine = EPServiceProviderManager.
getDefaultProvider();
EPRuntime runtimeEngine = engine.getRuntime();
...
LocationReport event = new LocationReport(assetId,x,y,zone);
runtimeEngine.sendEvent(event);

80. Event filtering
// Filter for location report by location rectangle
select * from LR(x in [4:10], y in [6:12])

81. Sliding windows and aggregation
// Count all assets reporting zone 10 in last 30 sec
select count(*) from LR(zone=10).win:time(30 sec)

82. Grouped windows and output rate limiting
// Get X location range of the last 100 events per zone
// every 1 minute
select zone, min(x), max(x)
from LR.std:groupby('zone').win:length(100 events)
output every 1 min

83. Continuous joins
// Fire when any asset enters zone 2 before zone 1
select Zone2.assetId
from
LR(zone=2).win:time(1 day) Zone2
left outer join
LR(zone=1).win:time(1 day) Zone1
on Zone1.assetId = Zone2.assetId
where Zone1.assetId is null

84. Historical or reference data
// Alert when we hit the minimum inventory
// for a given zone
select zone, count(*)
from LR.std:unique(assetId) as lr,
sql:db[select mini from Minimum where zone=${lr.zone}]
having mini < count(*)

85. Subquery
// Notify when the asset id is not known
select * from LR
where assetId not in
(select assetId
from KnownAsset.std:unique('assetId'))

86. Data window views
● win:length
● win:length_batch
● win:time
● win:time_batch
● win:time_length_batch
● win:time_accum
● win:ext_timed
● ext:sort_window
● ext:time_order

87. Data window views
● std:unique
● std:groupwin
● std:lastevent
● std:firstevent
● std:firstunique
● win:firstlength
● win:firsttime

88. Statistics views
● std:size
● stat:uni
● stat:linest
● stat:correl
● stat:weighted_avg

89. CEP Application Development Steps
● Define Event
● Define Event Type
● Define EPL Statements
● Publish EPL Statements
● Subscribe Output Adapter
● Attache Input Adapter

90. CEP Application Development Points
● Runtime 환경 결정
● Resilience를 지원 여부
● Zero-downtime event processing
● Support dynamic EPL Statement updates
● Input/Output Adapter에서 bottleneck이 발생
● 그러므로 Adapter의 성능이 중요
● Event의 타입과 용량에 따라서 성능에 영향
● Input Adapter의 통신 protocol 결정 중요
● CEP Engine Performance Tuning

91. CEP Application Key Considerations
● Low Latency
● High Throughput
● High Availability - Failover
● Zero-downtime

92. Performance
● Throughput
○ events/sec
● Latency
○ us/event
● 성능 측정은 throughput을 고정하고 latency statistics로 측정
● http://docs.codehaus.org/display/ESPER/Esper+performance

93. Latency statistics
● Throughput 995900 (active 0 pending 0)
---Stats - engine (unit: ns)
Avg: 2656 #9936457
0< 5000: 99.78% 99.78% #9914178
5000 < 10000: 0.20% 99.98% #20120
10000 < 15000: 0.01% 99.98% #646
15000 < 20000: 0.00% 99.99% #69
20000 < 25000: 0.00% 99.99% #36
25000 < 50000: 0.00% 99.99% #56
50000 < 100000: 0.00% 99.99% #26
100000 < 500000: 0.00% 99.99% #189
500000 < 1000000: 0.00% 99.99% #2
1000000 < 2500000: 0.00% 99.99% #68

94. Thanks
Q&A

95. Thanks
Q&A

96. Practice!!
http://goo.gl/5ijAu

97. JBoss Community (http://www.jboss.org)
Korea JBoss User Group (http://cafe.naver.com/jbossug)