How to leverage Kafka data streams with Neo4j

larus-ba.it/neo4j
@AgileLARUS
Andrea Santurbano / @santand84
#GraphRM
Roma, 12/07/2019
How to leverage Apache Kafka data streams
with Neo4j

LARUS Business Automation Srl Italy’s #1 Neo4j Partner
WHO AM I?
Andrea
[:WORKS_AT]
[:LOVES]
[:INTEGRATOR_LEADER_FOR]

WHO’S LARUS?
LARUS BUSINESS AUTOMATION
● Founded in 2004
● Headquartered in Venice, ITALY
● Delivering services Worldwide
● Mission: “Bridging the gap between Business and IT”
#1 Solution Partner in Italy since 2013
● Creator of the Neo4j JDBC Driver
● Creator of the Neo4j Apache Zeppelin Interpreter
● Creator of the Neo4j ETL Tool
● Developed 90+ APOC
VENICE
[:BASED_IN]

INTEGRATOR LEADERS FOR NEO4J
2016
Neo4j JDBC Driver
20152011
First Spikes
in Retail for
Articles’
Clustering
2014 2018
Neo4j APOC, ETL, Spark, Zeppelin, Kafka

WE ARE HIRING!
[:HIRES]
We’re looking for PASSIONATE java DEVELOPERS
to WORK on CHALLENGING PROJECTS
with CUTTING EDGE TECHNOLOGIES (such as Kafka and Neo4j)
(in Rome and Pescara)

larus-ba.it/neo4j
@AgileLARUS
Agenda

Agenda
● What is Neo4j Streams?
○ What is Apache Kafka?
○ How we combined Neo4j and Kafka?
● The Change Data Capture Module
○ DEMO
● The Streams Procedure
○ DEMO
● The Sink
○ DEMO

larus-ba.it/neo4j
@AgileLARUS
Enables Kafka Streaming on Neo4j!
What is Neo4j Streams?

What is Apache Kafka?
A DISTRIBUTED STREAMING PLATFORM
Has three key capabilities:
● Publish and subscribe to streams of records;
● Store streams of records in a fault-tolerant
durable way;
● Process streams of records as they occur.

HOW IT WORKS?
1. TOPICS: a topic is a category or feed name to
which records are published.
2. PARTITIONS: for each topic, the Kafka cluster
maintains a partitioned log

HOW IT’S USED?
Kafka is generally used for two classes of
applications:
● Building real-time streaming data pipelines;
● Building real-time streaming applications.

What is Neo4j Streams?
Andrea
[:AUTHOR_OF][:CREATOR_OF] X
Michael
ENABLES DATA STREAM ON NEO4J
The project is a Neo4j Plugin composed of several parts:
● Neo4j Streams Change Data Capture;
● Neo4j Streams Sink;
● Neo4j Streams Procedures
We also have a Kafka Connect Plugin:
● Kafka Connect Sink plugin.

larus-ba.it/neo4j
@AgileLARUS
Stream database changes!
Neo4j Streams: Change Data Capture

Change data “what”?
In databases, Change Data Capture (CDC) is a set of software design patterns used to determine (and
track) the data that has changed so an action can be taken using the changed data.
Well suited use-cases?
● CDC solutions occur most often in data-warehouse environments;
● Allows to replicate databases without having a/much performance impact on its operation.

How it works?
Each transaction communicates its changes to our event listener:
● exposing creation, updates and deletes of Nodes and Relationships
● providing before-and-after information
● configuring property filtering for each topic
Those events are sent asynchronously to Kafka, so the commit path should not be influenced by that.

larus-ba.it/neo4j
@AgileLARUS
DEMO

larus-ba.it/neo4j
@AgileLARUS
Interact with Apache Kafka directly from Cypher!
Neo4j Streams: Procedures

Neo4j Streams: Streams Procedures
CONSUME/PRODUCE DATA DIRECTLY FROM CYPHER
The Neo4j Streams project comes out with two procedures:
● streams.publish: allows custom message streaming from Neo4j to the conﬁgured environment by
using the underlying conﬁgured Producer;
● streams.consume: allows consuming messages from a given topic.

larus-ba.it/neo4j
@AgileLARUS
Neo4j Streams: Streams Procedures
DEMO

larus-ba.it/neo4j
@AgileLARUS
Ingest data into Neo4j directly from the Stream!
Neo4j Streams: Sink

Neo4j Streams: Sink
INGEST YOUR DATA, WITH YOUR RULES
The sink provides several ways in order to ingest data from Kafka:
● Via Cypher Template
● Via CDC event published by another Neo4j Instance via the CDC module
● Via projection of a JSON event into Node/Relationship by providing an extraction pattern

Neo4j Streams: Sink
INGEST YOUR DATA, WITH YOUR RULES
Initially, we thought about a generic consumer with a ﬁxed projection of events into Nodes and
Relationships.
We decided that instead, we want to give the user the power to use custom Cypher statements per topic
to turn Events into arbitrary graph structures.
So you can choose by yourself what to do with a complex Kafka event. Which parts of it you want to use
for which purpose.

Neo4j Streams: Sink
INGESTION VIA CYPHER TEMPLATE
Besides your Kafka connection information, you just add entries like this to your Neo4j config.
streams.sink.topic.cypher.<TOPIC>=<CYPHER_QUERY>
For example:
streams.sink.topic.cypher.my-topic=MERGE (n:Label {id: event.id}) ON CREATE
SET n += event.properties
Under the hood, the consumer takes a batch of Events and passes them as $batch parameter to the
Cypher statement, which we prefix with an UNWIND, so each individual entry is available as èvent`
identifier to your statement. So the final statement executed by us would look like this:
UNWIND $batch AS event
MERGE (n:Label {id: event.id})
ON CREATE SET n += event.properties

Neo4j Streams: Sink
INGESTION VIA CDC EVENT FROM ANOTHER NEO4J INSTANCE
We allow ingesting the data in two ways:
● The SourceId strategy which merges the nodes/relationships by the CDC event ìd` field (it's related to
the Neo4j physical ID)
streams.sink.topic.cdc.sourceId=<TOPICS_SEPARATED_BY_SEMICOLON>
● The Schema strategy which merges the nodes/relationships by the constraints (UNIQUENESS,
NODE_KEY) defined in your graph model
streams.sink.topic.cdc.schema=<TOPICS_SEPARATED_BY_SEMICOLON>

Neo4j Streams: Sink
INGESTION VIA JSON PROJECTION
You can extract nodes and relationships from a JSON by providing a extraction pattern.
Each property can be preﬁxed with:
● !: identify the id (could be more than one property), it's *mandatory*
● -: exclude the property from the extraction
● Labels can be chained via :
Tombstone Record Management
This ingestion strategy come out with the support to the Tombstone Record, in order to leverage it your
event should contain as key the record that you want to delete and `null` for the value.

Neo4j Streams: Sink
INGESTION VIA JSON PROJECTION - NODE PATTERN EXTRACTION
Given:
{"userId": 1, "name": "Andrea", "surname": "Santurbano", "address": {"city": "Venice", "cap": "30100"}}
You can transform it into a node by specifying one of these patterns:
● User:Actor{!userId} or User:Actor{!userId,*} => (User:Actor{userId: 1, name: 'Andrea', surname:
'Santurbano', àddress.city`: 'Venice', àddress.cap`: 30100})
● User{!userId, surname} => (User:Actor{userId: 1, surname: 'Santurbano'})
● User{!userId, surname, address.city} => (User:Actor{userId: 1, surname: 'Santurbano', àddress.city`:
'Venice'})
● User{!userId,-address} => (User:Actor{userId: 1, name: 'Andrea', surname: 'Santurbano'})

Neo4j Streams: Sink
INGESTION VIA JSON PROJECTION - RELATIONSHIP PATTERN EXTRACTION
Given:
{"userId": 1, "productId": 100, "price": 10, "currency": "€", "shippingAddress": {"city": "Venice", cap: "30100"}}
You can transform it into a relationship by specifying one of these patterns:
● (User{!userId})-[:BOUGHT]->(Product{!productId}) or (User{!userId})-[:BOUGHT{price,
currency}]->(Product{!productId}) => (User{userId: 1})-[:BOUGHT{price: 10, currency: '€',
`shippingAddress.city`: 'Venice', `shippingAddress.cap`: 30100}]->(Product{productId: 100})
● (User{!userId})-[:BOUGHT{price}]->(Product{!productId}) => (User{userId: 1})-[:BOUGHT{price:
10}]->(Product{productId: 100})

Neo4j Streams: Sink
HOW WE MANAGE BAD DATA
The Neo4j Streams Sink module provide a Dead Letter Queue mechanism that if activated re-route all
“bad-data” to a conﬁgured topic.
What we mean for “bad-data”?
● De-Serialization errors. I.e. bad formatted JSON:
{id: 1, "name": "Andrea", "surname": "Santurbano"}
● Transient errors while ingesting data into the DB.

Neo4j Streams: Kafka Connect Sink
WHAT IS KAFKA CONNECT?
In open source component of Apache Kafka, is a
framework for connecting Kafka with external
systems such as databases, key-value stores,
search indexes, and file systems.
HOW IT WORKS?
It works exactly in the same way as the Neo4j Sink
plugin so you can provide for each topic your own
Cypher query.
You can download it from the Confluent HUB!
And it has the Verified GOLD badge!

larus-ba.it/neo4j
@AgileLARUS
Real-time Polyglot Persistence with
Elastic, Kafka and Neo4j
DEMO

RT Polyglot Persistence with Elastic, Kafka & Neo4j

Neo4j Streams: Lessons learned
THE POWER OF THE STREAM!
● We have seen how to use the CDC in order to
stream transaction events from Neo4j to other
systems;
● We have seen how to use the SINK in order to
ingest data into Neo4j by providing our own
business rules;
● We have seen how to use the Streams
PROCEDURES in order to consume/produce
data directly from Cypher.

larus-ba.it/neo4j
@AgileLARUS
Community feedback & Beyond Kafka
What’s next?

GIVE US FEEDBACK
PROVIDE US FEEDBACK
If you plan to use the Streams Plugin please give us a feedback!
https://github.com/neo4j-contrib/neo4j-streams

CODE REPOSITORY
https://github.com/conker84/kafka-rome-june-2k19

larus-ba.it/neo4j
@AgileLARUS
THANKS!
#GraphRM
Roma, 12/07/2019
Andrea Santurbano
andrea.santurbano@larus-ba.it / @santand84

How to leverage Kafka data streams with Neo4j

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à How to leverage Kafka data streams with Neo4j

Similaire à How to leverage Kafka data streams with Neo4j (20)

Plus de GraphRM

Plus de GraphRM (14)

Dernier

Dernier (20)

How to leverage Kafka data streams with Neo4j