ODSC - Batch to Stream workshop - integration of Apache Spark, Cassandra, PostgreSQL, Redpanda, Debezium, and Benthos to master building advanced real-time data pipelines.

•

0 j'aime•15 vues

The slide that goes with the workshop. - Demonstrate Transitioning to Real-Time Data Systems: This track shows how to transition from traditional batch processing to real-time data handling, emphasizing the critical importance of timely updates in scenarios like air traffic control where delays can be hazardous. - Implement Real-Time Data Solutions: Participants learn to implement and integrate a technology stack featuring Apache Spark, Cassandra, and PostgreSQL to effectively manage and process live data feeds from various sources, suitable for high-stakes environments. - Employ Streaming and Data Capture Technologies: The workshop demonstrates how to utilize streaming platforms and Change Data Capture (CDC) with tools like Redpanda and Debezium to ensure continuous data flow and immediate update capture, streamlining operations and data accuracy. - Apply Data Querying Techniques: It concludes by teaching how to apply advanced querying methods using Flink’s SQL client to incorporate real-time data queries into existing systems, ensuring efficient data manipulation and integration into relational and NoSQL databases.

Logiciels

B A T C H
T O
S T R E A M
D A T A P I P E L I N E
Christina Lin
Redpanda
Developer Advocate

Agenda
• Quick Intro – 10-15 mins
• HANDS-ON – 35-40 mins
• Q & A – 5-10 mins
© 2024 REDPANDA DATA

Christina Lin
Developer Advocate, Redpanda
aka. The Redpanda Lady
SOA
WebSphere
DB2
Sybase
Oracle
MQ
J2EE
EJB
DevOps
Microservice
EIP
K8s
Agile
Integration
Data
Mesh
Active MQ
Living data stack
Resilience - handle failures and scale gracefully
Elasticity – infrastructure that can scale dynamically
Decentralization - data ownership, empowering
individual teams
Performance - low latency and high throughput
Autonomy – self service, define quality, and access
Nimble - efficient data movement
Distributed -distributed data processing for cloud native
Agility – quickly respond to change in data
© 2024 REDPANDA DATA

© 2024 REDPANDA DATA
An ordinary day of Data Engineer

© 2024 REDPANDA DATA
© 2024 REDPANDA DATA
Stateless
Streaming Pipeline
Transform
format Change, masking, filtering, validating
Dispatch, Wiretap
Spilt, multiple destination
Control
reroute
Normalize/ Denormalize Enrich
Multiple ingestion
Stateful
Streaming Pipeline
Complex event processing
Time-window based processing
Enrich
Multiple ingestion
Micro batch Pipeline
Transform for large output (Dataset)
Partitioning Split workload
Analytics
batch
Pipeline
Analytics large volume (legacy)
Transform large output (Dataset, legacy)
Transport large unstructured data
Better scalability for pipelines

Batch
CSV
Every 10 mins
CSV
Right away!
CSV
Stream
Batch
pipeline
Batch Processing
Batch
pipeline
TheWorkshop overview
© 2024 REDPANDA DATA
https://bit.ly/odsc-redpanda

VM
postgres:5432 cassandra:5432
postgresload.ipynb
© 2024 REDPANDA DATA
https://bit.ly/odsc-redpanda
The Batch
cassandra:5432

VM
postgres:5432 cassandra:5432
cassandraload.ipynb
© 2024 REDPANDA DATA
https://bit.ly/odsc-redpanda
The Batch

VM
postgres:5432 cassandra:5432
spark.ipynb
© 2024 REDPANDA DATA
https://bit.ly/odsc-redpanda
The Batch
CSV
CSV
CSV

VM
postgres:5432 cassandra:5432
Map.ipynb
© 2024 REDPANDA DATA
https://bit.ly/odsc-redpanda
Dashboard View
CSV
CSV
CSV

VM
postgres:5432cassandra:5432
redpanda-0:9092
© 2024 REDPANDA DATA
https://bit.ly/osdc-redpanda
Let’s Stream

VM
postgres:5432cassandra:5432
© 2024 REDPANDA DATA
https://bit.ly/osdc-redpanda
Let’s Stream

VM
postgres:5432cassandra:5432
Config
© 2024 REDPANDA DATA
https://bit.ly/odsc-redpanda
Let’s Stream

VM
postgres:5432cassandra:5432
Job manager
Flink Data
Stream
Java JAR
© 2024 REDPANDA DATA
https://bit.ly/odsc-redpanda
Let’s Stream

VM
postgres:5432cassandra:5432
Job manager
CSV
© 2024 REDPANDA DATA
https://bit.ly/odsc-redpanda
Let’s Stream

VM
postgres:5432cassandra:5432
Job manager
CSV
© 2024 REDPANDA DATA
https://bit.ly/odsc-redpanda
Let’s Stream
CSV
CSV

VM
postgres:5432cassandra:5432
SQL Client
SQL
© 2024 REDPANDA DATA
https://bit.ly/odsc-redpanda
The Batch
CSV
CSV
CSV

Batch
CSV
Batch
pipeline
Batch
pipeline
Batch Processing
Every 10 mins
CSV
Right away!
CSV
Stream
© 2024 REDPANDA DATA
https://bit.ly/odsc-redpanda
TheWorkshop overview

© 2024 REDPANDA DATA
Stateless
Streaming Pipeline
Transform
format Change, masking, filtering, validating
Dispatch, Wiretap
Spilt, multiple destination
Control
reroute
Normalize/ Denormalize Enrich
Multiple ingestion
Stateful
Streaming Pipeline
Complex event processing
Time-window based processing
Enrich
Multiple ingestion
Better scalability for pipelines

© 2024 REDPANDA DATA
Keep Learning
Streaming - Communication
Basics of K8s networking
Connectivity
Performance
Docs
Get a peak under the hood.
https://docs.redpanda.com/
Blogs
Keep up to date with Redpanda.
https://redpanda.com/blog
Slack
Engage with our community.
https://redpanda.com/slack
Code
Check out the source.
https://github.com/redpanda-data
Redpanda University
Free, self-paced online learning
https://university.redpanda.com

Contenu connexe

Similaire à ODSC - Batch to Stream workshop - integration of Apache Spark, Cassandra, PostgreSQL, Redpanda, Debezium, and Benthos to master building advanced real-time data pipelines.

To stay competitive in today's global economy, organizations need to harness data from multiple sources, extract information and then make real-time decisions. Depending on the industry and the organization, Big Data encompasses information from multiple internal and external sources. Capturing, integrating and preparing this data for downstream analysis is often time-consuming and presents a big challenge. Today, organizations are struggling to cobble together different open source software to build an effective data pipeline that can withstand the volume as well as the speed of data ingestion and analysis. Robin relieves customers from the pains of building and maintaining a data pipeline and helps enterprises to make the most of Big Data. In this presentation, we will focus on how Robin’s containerization platform can be used to: - Build an agile and elastic data pipeline - Deploy, scale, and manage the most complex big data applications with just a single click of a button - Deal with variety, velocity, and volume of enterprise Big Data

Build agile and elastic data pipeline

Deba Chatterjee

Visual Mapping of Clickstream Data

DataWorks Summit

EsgynDB: A Big Data Engine. Simplifying Fast and Reliable Mixed Workloads

Srikanth Ramakrishnan

To remain competitive, organizations need to democratize access to fast analytics, not only to gain real-time insights on their business but also to power smart apps that need to react in the moment. In this session, you will learn how Kafka and SingleStore enable modern, yet simple data architecture to analyze both fast paced incoming data as well as large historical datasets. In particular, you will understand why SingleStore is well suited process data streams coming from Kafka.

SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...

HostedbyConfluent

Riverbed Granite

CTI Group

Steelhead DX for Datacenter-to-Datacenter optimization

Riverbed Technology

Building scalable data with kafka and spark

babatunde ekemode

Wasp2 - IoT and Streaming Platform

Paolo Platter

Riverbed Within Local Gov

michaelking

6WINDGate™ - High Performance Networking for Data Centers

6WIND

SimplifyStreamingArchitecture

Maheedhar Gunturu

High performance Spark distribution on PKS by SnappyData

VMware Tanzu

High performance Spark distribution on PKS by SnappyData

Carlos Andrés García

SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...

HostedbyConfluent

Not all workloads allow cloud computing. Low latency, cybersecurity, and cost-efficiency require a suitable combination of edge computing and cloud integration. This session explores architectures and design patterns for software and hardware considerations to deploy hybrid data streaming with Apache Kafka anywhere. A live demo shows data synchronization from the edge to the public cloud across continents with Kafka on Hivecell and Confluent Cloud.

Kafka for Real-Time Replication between Edge and Hybrid Cloud

Kai Wähner

Virtualized Big Data Platform at VMware Corp IT @ VMWorld 2015

Rajit Saha

[OpenStack Day in Korea 2015] Track 2-3 - 오픈스택 클라우드에 최적화된 네트워크 가상화 '누아지(Nuage)'

OpenStack Korea Community

#3 rsp success card

chanwitcs

Navigate Data Service using AWS

Arno Broekhof

This hands on workshop for OpenContrail will be led by Sreelakshmi Sarva & Aniket Daptari. This is a labs session so we will have hard RSVP limits. Please RSVP only if you are confident that you will be able to attend. About Sreelakshmi Sarva Sree is currently working as part of solution engineering team at Juniper’s Contrail team. She is responsible for delivering & managing SDN solutions & partnerships relating to Contrail. She has been with Juniper for the last 13 years working on various Routing, Switching, Network programmability & virtualization platforms. Prior to Juniper, She worked at Nortel networks in the Systems Engineering group. Sree received her Masters in Computer Science from University of Texas at Dallas and Bachelor’s in Computer Science from India. About Aniket Daptari Aniket is currently working as part of Juniper Networks' Contrail Cloud Solutions team. He is responsible for delivering SDN solutions and technology partnerships related to Contrail. He has been with Juniper for the last 3 years working on various Network programmability & virtualization platforms. Prior to Juniper, he worked at Cisco Systems in the Internet Systems Business Unit (Catalyst 6500). Aniket received his Masters in Computer Science from University of Southern California and a graduate certificate in Management Science and Engineering from Stanford University. Course Abstract This session will be the first of a series of OpenContrail hands-on tutorials for developers who want to get deep into OpenContrail code. This “Basic OpenContrail Programming” Hands-on Session will focus on making developers proficient in writing and contributing code for our OpenContrail Project. Session will cover the following areas 1) Contrail Overview · Use Cases · Architecture recap 2) Contrail Hands on · Demo + Hands on - Configuration , VN, VM, Network Policies etc · DevStack introduction

Banv meetup-contrail

nvirters

Similaire à ODSC - Batch to Stream workshop - integration of Apache Spark, Cassandra, PostgreSQL, Redpanda, Debezium, and Benthos to master building advanced real-time data pipelines. (20)

Build agile and elastic data pipeline

Visual Mapping of Clickstream Data

EsgynDB: A Big Data Engine. Simplifying Fast and Reliable Mixed Workloads

SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...

Riverbed Granite

Steelhead DX for Datacenter-to-Datacenter optimization

Building scalable data with kafka and spark

Wasp2 - IoT and Streaming Platform

Riverbed Within Local Gov

6WINDGate™ - High Performance Networking for Data Centers

SimplifyStreamingArchitecture

High performance Spark distribution on PKS by SnappyData

SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...

Kafka for Real-Time Replication between Edge and Hybrid Cloud

Virtualized Big Data Platform at VMware Corp IT @ VMWorld 2015

[OpenStack Day in Korea 2015] Track 2-3 - 오픈스택 클라우드에 최적화된 네트워크 가상화 '누아지(Nuage)'

#3 rsp success card

Navigate Data Service using AWS

Banv meetup-contrail

Plus de Christina Lin

Bangalore Meetup - Enable realtime machine learning with streaming data

Christina Lin

Kafka summit apac session

Christina Lin

Serverless integration anatomy

Christina Lin

Day in the life event-driven workshop

Christina Lin

Agile integration cloud native developement

Christina Lin

Dev conf .in cloud native reference architecture .advance

Christina Lin

Camel k Taiwan Java user group

Christina Lin

Devoxxma-API centric microservices Architecture

Christina Lin

JBoss Fuse - Fuse workshop EAP container

Christina Lin

Red Hat Tech Exchange Integration often involves storing, retrieving, and transforming data. Using a traditional database in your integration is likely to becomes a bottleneck that is expensive and hard to scale. By combining the agile, lightweight, Enterprise Integration pattern (EIP) based integrations using open source technology with scalable in-memory storage, it’s possible to achieve near linear scalability and boost performance of your integration services.

Supercharge Your Integration Services

Christina Lin

Improve business process with microservice integration

Christina Lin

Integrating BPM with Fuse

Christina Lin

Scalable Integration with JBoss Fuse

Christina Lin

JBoss Fuse - Fuse workshop Error Handling

Christina Lin

JBoss Fuse Workshop 101 part 6

Christina Lin

JBoss Fuse Workshop 101 part 5

Christina Lin

JBoss Fuse Workshop 101 part 4

Christina Lin

JBoss Fuse Workshop 101 part 3

Christina Lin

JBoss Fuse Workshop 101 part 2

Christina Lin

Jboss Fuse Workshop 101 part 1

Christina Lin

Plus de Christina Lin (20)

Bangalore Meetup - Enable realtime machine learning with streaming data

Kafka summit apac session

Serverless integration anatomy

Day in the life event-driven workshop

Agile integration cloud native developement

Dev conf .in cloud native reference architecture .advance

Camel k Taiwan Java user group

Devoxxma-API centric microservices Architecture

JBoss Fuse - Fuse workshop EAP container

Supercharge Your Integration Services

Improve business process with microservice integration

Integrating BPM with Fuse

Scalable Integration with JBoss Fuse

JBoss Fuse - Fuse workshop Error Handling

JBoss Fuse Workshop 101 part 6

JBoss Fuse Workshop 101 part 5

JBoss Fuse Workshop 101 part 4

JBoss Fuse Workshop 101 part 3

JBoss Fuse Workshop 101 part 2

Jboss Fuse Workshop 101 part 1

Dernier

A Deep Dive into Secure Product Development Frameworks.pdf

ICS

Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...

Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg

Abortion Clinic in Bloemfontein [(+27832195400*)]🏥Safe Abortion Pills In Bloe...

Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg

Abortion Pill Prices Germiston ](+27832195400*)[ 🏥 Women's Abortion Clinic in...

Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg

Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...

Abortion Clinic

Abortion Clinic In Pongola ](+27832195400*)[ 🏥 Safe Abortion Pills In Pongola...

Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg

Abortion Clinic In Stanger ](+27832195400*)[ 🏥 Safe Abortion Pills In Stanger...

Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg

Evolving Data Governance for the Real-time Streaming and AI Era

confluent

Abortion Pill Prices Polokwane ](+27832195400*)[ 🏥 Women's Abortion Clinic in...

Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg

Talk given at NewCrafts Paris Conference 2024 Abstract: Have you ever observed quality problems with what your team is delivering? Have you ever felt your team is learning about the actual status quo late, too late to include these insights in your next steps? Have you ever wondered how to influence change when you’re “just” a specialist on the team, maybe even felt powerless? I’ve mostly worked with people who never experienced effective collaboration across roles and specialties. How to get such a team to move toward testing throughout the delivery workflow from idea to production, on all kinds of levels, for all parts of the product, by the whole team? How to consider quality from the start and have it continuously inform our decision-making? Over the years, I’ve built a repertoire of tactics to help teams transform toward holistic testing and quality. They range from meeting people where they are to creating guidelines, pairing and many others. It’s a toolbox of approaches to try out and find the sweet spot where people start moving toward better directions together. These tactics are fallible, not all of them work in all contexts, and yet they have helped my teams get to a better place. Join this talk to get inspired to take things into your hands. Learn what you can try yourself in your context to help your team make the first steps on your journey towards holistic testing and quality. Key learnings: - Learn how you can influence team culture, no matter your role or experience level - Add team transformation tactics to your toolbox of approaches to try - Understand the many ways we can move towards holistic testing and quality

Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...

Lisi Hocke

Test Automation Design Patterns_ A Comprehensive Guide.pdf

kalichargn70th171

Unlock the full potential of your software development lifecycle with SpiraPlan's powerful REST APIs. Led by experts from Inflectra and our partner, MVerve, this webinar offers a practical guide to leveraging the API's power for automation, integration, and data-driven insights. Whether you're an API expert or beginner, our recap blog is packed with valuable information to enhance your SpiraPlan experience. The webinar emphasized how SpiraPlan's REST API allows you to: • Automate Workflows: Streamline your processes by automating repetitive tasks. • Integrate with Existing Tools: Seamlessly connect SpiraPlan with your existing software ecosystem. • Build Custom Solutions: Tailor SpiraPlan to your unique needs by developing custom applications and integrations. • Gain Deeper Insights: Extract valuable data from SpiraPlan for advanced reporting and analysis.

From Theory to Practice: Utilizing SpiraPlan's REST API

Inflectra

Weeding your micro service landscape.pdf

timtebeek1

This session unveils the multifaceted horizontal scaling strategies that power Wix's robust infrastructure. From Kafka consumer scaling and dynamic traffic routing for site segments to DynamoDB sharding and MySQL clusters custom routing with ProxySQL, we dissect the mechanisms that ensure scalability and performance at Wix. Attendees will learn about the art of sharding and routing key selection across different systems, and how to apply these strategies to their own infrastructure. We'll share insights into choosing the right scaling strategy for various scenarios, balancing between managed services and custom solutions. Key Takeaways: - Grasp various sharding techniques and routing strategies used at Wix. - Understand key considerations for sharding key and routing rule selection. - Learn when and why to choose specific horizontal scaling strategies. - Gain practical knowledge for applying these strategies to achieve scalability and high availability. Join us to gain a blueprint for scaling your systems horizontally, drawing from Wix's proven practices.

Effective Strategies for Wix's Scaling challenges - GeeCon

Natan Silnitsky

Abortion Clinic in Midrand [(+27832195400*)]🏥Safe Abortion Pills In Midrand | Abortions Clinic | Quality & Affordable Healthcare Services !! Abortion Clinic Near Me Cost, Price, Women's Clinic Near Me, Abortion Clinic Near, Abortion Doctors Near me, Abortion Services Near Me, Abortion Pills Over The Counter, Abortion Pill Doctors' Offices, Abortion Clinics, Abortion Places Near Me, Cheap Abortion Places Near Me, Medical Abortion & Surgical Abortion, approved cyctotec pills and womb cleaning pills too plus all the instructions needed This Discrete women’s Termination Clinic offers same day services that are safe and pain free, we use approved pills and we clean the womb so that no side effects are present. Our main goal is that of preventing unintended pregnancies and unwanted births every day to enable more women to have children by choice, not chance. We offer Terminations by Pill and The Morning After Pill.” Our Private VIP Abortion Service offers the ultimate in privacy, efficiency and discretion. we do safe and same day termination and we do also womb cleaning as well its done from 1 week up to 28 weeks. We do delivery of our services world wide SAFE ABORTION CLINICS/PILLS ON SALE WE DO DELIVERY OF PILLS ALSO Abortion clinic at very low costs, 100% Guaranteed and it’s safe, pain free and a same day service. It Is A 45 Minutes Procedure, we use tested abortion pills and we do womb cleaning as well. Alternatively the medical abortion pill and womb cleansing !!!

Abortion Clinic in Midrand [(+27832195400*)]🏥Safe Abortion Pills In Midrand |...

Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg

“I think, therefore I am.” From the moment we arrived on this chaotic planet, we began learning (even before that, actually, but let’s ignore that for now). Every day, we learn: good patterns, bad patterns—the brain doesn’t differentiate! However, for us to simply say, “I learn while I do,” might be a bit too easy. Or, at least, one might wonder if it’s the most efficient and satisfying way to learn. During this talk, you'll learn how our brains absorb and retain new information. Afterward, we’ll explore how you can create the best conditions and environment for your specific learning needs. Among other things, we’ll cover what has scientifically been proven to be the ideal learning blocks regarding time and content. We’ll discover how to create an environment that improves our focus and remain in the “flow.” And we’ll discuss the profound impact of AI on the entire learning landscape. As we navigate the ever-changing landscape of software development, this knowledge will empower you to enhance your own learning curve and, just as importantly, extend a helping hand to those on their own learning path.

Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024

SimonedeGijt

Welcome the the next GRCPP meetup where we will introduce a powerful new feature in C++20: Concepts. Concepts are a way to specify requirements on template parameter types and to make templates more expressive and easier to understand.. We will show how to use concepts to create "interfaces" for your template types, how to simplify your code and provide better error messages. We will demonstrate how to use `requires` to create "contracts" for your templates and how to use `concept` to constrain the acceptable types.

[GRCPP] Introduction to concepts (C++20)

Dimitrios Platis

Abortion Pill Prices Jane Furse ](+27832195400*)[🏥Women's Abortion Clinic in Jane Furse ● Abortion Pills For Sale in Jane Furse ● Jane Furse 🏥🚑!! Abortion Clinic Near Me Cost, Price, Women's Clinic Near Me, Abortion Clinic Near, Abortion Doctors Near me, Abortion Services Near Me, Abortion Pills Over The Counter, Abortion Pill Doctors' Offices, Abortion Clinics, Abortion Places Near Me, Cheap Abortion Places Near Me, Medical Abortion & Surgical Abortion, approved cyctotec pills and womb cleaning pills too plus all the instructions needed This Discrete women’s Termination Clinic offers same day services that are safe and pain free, we use approved pills and we clean the womb so that no side effects are present. Our main goal is that of preventing unintended pregnancies and unwanted births every day to enable more women to have children by choice, not chance. We offer Terminations by Pill and The Morning After Pill.” Our Private VIP Abortion Service offers the ultimate in privacy, efficiency and discretion. we do safe and same day termination and we do also womb cleaning as well its done from 1 week up to 28 weeks. We do delivery of our services world wide SAFE ABORTION CLINICS/PILLS ON SALE WE DO DELIVERY OF PILLS ALSO Abortion clinic at very low costs, 100% Guaranteed and it’s safe, pain free and a same day service. It Is A 45 Minutes Procedure, we use tested abortion pills and we do womb cleaning as well. Alternatively the medical abortion pill and womb cleansing !!

Abortion Pill Prices Jane Furse ](+27832195400*)[🏥Women's Abortion Clinic in ...

Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg

Novo Nordisk: When Knowledge Graphs meet LLMs

Neo4j

Navigation in flutter – how to add stack, tab, and drawer navigators to your ...

Flutter Agency

Dernier (20)