How to handle time series data with OrientDB

1. Time flows, on Graph
Managing event sequences and time series with a Document-
Graph Database
FOSDEM 2015
Enrico Risa
Orient Technologies LTD
Twitter: @wolf4ood
Emanuele Tagliaferri
Orient Technologies LTD
Twitter: @tglman

2. Time What…?
Time series:
A time series is a sequence of data points, typically
consisting of successive measurements made over a
time interval (Wikipedia)

3. Time What…?
Event sequences:
• A set of events with a timestamp
• A set of relationships “happened
before/after”
• Cause and effect relationships

4. Graph approaches
•. Nodes/Edges
•. Index free adjacency
•. Fast traversal
•. Dynamic structure

5. Graph approaches
Linked sequence
e1e1 e2e2
next
e3e3
next
e4e4
next
e5e5
next
(timestamp on vertex)

6. Graph approaches
linked sequence (tag based)
e1e1 e2e2
nextTag1
e3e3
nextTag2
e4e4
nextTag1
e5e5
nextTag1
nextTag2
[Tag1, Tag2] [Tag1]
[Tag1, Tag2]
[Tag1]
[Tag2]

7. Graph approaches
Hierarchy
e1e1 e2e2 e6
0
e6
0
11
11
88
2424
22 6060…
…
Days
Hours
Minutes
Seconds
…
e3e3

8. Graph approaches
Mixed
e1e1 e2e2 e6
0
e6
0
11
11
88
2424
22 6060…
…
Days
Hours
Minutes
Seconds
…
e3e3

9. Current approaches
Advantages
•. Flexible
•. Events can be connected together in different ways
•. You can navigate events following a path by time or
tag.

10. Current approaches
Disadvantages
•. Slow query for a high number of event

11. Optimization
● Data Pre-Aggregation

12. Optimization
Pre-aggregate
11
11
88
2424
22 6060…
Days
Hours
Minutes
…
Graph

13. Optimization
Pre-aggregate
11
11
88
2424
22 6060…
Days
Hours
Minutes
…
Graph
sum()

14. Optimization
Pre-aggregate
11
11
88
2424
22 6060…
Days
Hours
Minutes
…
Graph
sum()
sum()

15. Optimization
Aggregation logic
• Second 0 -> insert
• Second 1 -> insert
• …
• Second 57 -> insert
• Second 58 -> insert
• Second 59 -> insert + aggregate update
– Write aggregate value on minute vertex
●
Minute == 59? Calculate aggregate on hour vertex

16. OrientDB
How to aggregate
Hooks: Server side triggers (Java or Javascript),
executed when DB operations happen (eg. Insert or
update)
Java interface:
Public RESULT onBeforeInsert(…);
public void onAfterInsert(…);
public RESULT onBeforeUpdate(…);
public void onAfterUpdate(…);

17. Optimization
11
11
88
2424
22 6060…
Days
Hours
Minutes
…
sum = 1000
sum = 15000
sum = 300
incomplete
complete
11 22
sum = null
sum = null

18. Optimization
Query logic:
• Traverse from root node to specified level
(filtering based on vertex data)
• Is there aggregate value?
– Yes: return it
– No: go one level down and do the same
Aggregation on a level will be VERY fast if you
have horizontal edges!

19. OrientDB
How to calculate aggregate values with a query
Input params:
- Root node (suppose it is #11:11)
select sum(aggregateVal) from (
traverse out() from #11:11
while in().aggregateVal is null
)
With the same logic you can query based on time
windows

20. Time Series
Proof of Concept

21. POC Implementation
Core:
● As OrientDB Plugin
● Rely on Hooks
● Aggregation Engine
● Handle all Time Unit
Data Visualization:
● Simple UI (Realtime/History)
● Query in Studio

22. Core
● Plugin that register hook and some input/output
source (websocket ,message queue, socket etc..)
● Hook on Event Class (entry point)
- Event can be saved or not.
- Aggregations are made when the lower time units
changes
- Pre-allocation of TimeUnit Pointers
● Time unit tracked:
-Year
-Month
-Day
-Minute
-Second

23. Core
Advantages
● Simple (Few lines of code)
● No Indexes
● Easy to use
– Plain OrientDB sql to insert an event
insert into event set bets = 1, cpu = 50
● Fast (Especially in plocal mode)

24. Core
Disadvantages
● Too Simple (For now)
● Aggregator hardcoded (Maybe
javascript aggregator?)

25. Data Visualization
Two Charts:
● Realtime data through WebSocket
The engine pushes the events received every seconds
● Range query for history Data
Using the powerfull array range notation we can query for
a specific time range

26. Let's Run It

27. Data Query Time unit
● Array Notation
select
expand(m[1].d[30].h[13].m[5-10])
from year where time = 2015
● Traverse with Next
traverse next from
(select expand(m[1].d[26].h[19].m[37])
from year where time = 2015 )
while $depth <= 3

28. Data Query Aggregation
● Array Notation
select sum(bets)from (select
expand(m[1].d[30].h[13].m[5-10])
from year where time = 2015)
● Traverse with Next
select sum(bets)from {traverse next from
(select expand(m[1].d[26].h[19].m[37])
from year where time = 2015 )
while $depth <= 3)

29. Multi-Model Optimization!
We got OrientDB

30. • Document database (schema-free, complex
properties)
• Graph database (index-free adjacency, fast traversal)
• SQL (extended)
• Operational (schema - ACID)
• OO concepts (Classes, inheritance, polymorphism)
• REST/JSON interface
• Native Javascript (extend query language, expose
services, event hooks)
• Distributed (Multi-master replica/sharding)
architecture

31. ●
Studio 2.0
●
Lucene & ETL in bundle
●
WAL management (Fuzzy Checkpoint)
●
Schema Driven Serialization
●
Autosharding strategy on Distributed

32. OrientDB
First step: put them together
11
11
88
2424
22 6060…
Days
Hours
Minutes
…
{
0: 1000,
1: 1500.
…
59: 96
}

33. OrientDB
First step: put them together
11
11
88
2424
22 6060…
Days
Hours
Minutes
…
{
0: 1000,
1: 1500.
…
59: 96
}
<- IT’S A VERTEX TOO!!!
Graph
Document

34. OrientDB
put them together
11
88
2424
Days
Hours…
{
0: {
0: 1000,
1: 1500,
…
59: 210
}
1: { … }
…
59: { … }
}
Graph
Document

35. Where should I stop?
It depends on my domain and
requirements.

36. OrientDB
Third step: Complex domains
11
11 22 6060…
Hours
Minutes
{
0: {val: 1000},
1: {val: 1500}.
…
59: {
val: 96,
eventTags: [tag1, tag2]
…
}
}
Graph
Document <- Enrich the domain

37. One model is not enough
One of most common issues of my customers
is:
“I have a zoo of technologies in my application
stack, and it’s getting worse every day”
My answer is: Multi-Model DB
of course ;-)

38. Thank you!
Enrico Risa
Orient Technologies LTD
Twitter: @wolf4ood
Emanuele Tagliaferri
Orient Technologies LTD
Twitter: @tglman