Live Demo Jam Expands: The Leading-Edge Streaming Data Platform with NiFi, Kafka, and Flink

•

1 j'aime•527 vues

Live Demo Jam Expands: The Leading-Edge Streaming Data Platform with NiFi, Kafka, and Flink Timothy Spann Twitter - @PaasDev // Blog: www.datainmotion.dev Frequent speaker at major conferences and events. Principal DataFlow Field Engineer for streaming around Apache NiFi, NiFi Registry, MiNiFi, Kafka, Kafka Connect, Kafka Streams, Flink, Flink SQL, SMM, SRM, SR and EFM. Previously at E&Y, HPE, Pivotal & Hortonworks Question #1 What is the most difficult part of an Edge Flow? Gateway Agent Edge Data Collection Processing Data https://github.com/tspannhw/DemoJam2021 https://github.com/tspannhw/CloudDemo2021

Logiciels

Live Demo Jam Expands: The Leading-Edge
Streaming Data Platform with NiFi, Kafka, and
Flink
Live demo by Tim Spann, Principal DataFlow Field Engineer

© 2021 Cloudera, Inc. All rights reserved. 2
Housekeeping items for today
• Ask your questions via the Q&A widget
• Select “YES” to the exit survey to be contact by Cloudera
• Check out additional resources to learn more

© 2021 Cloudera, Inc. All rights reserved. 3
Who?
Timothy Spann
Twitter - @PaasDev // Blog: www.datainmotion.dev
Frequent speaker at major conferences and events.
Principal DataFlow Field Engineer for streaming around
Apache NiFi, NiFi Registry, MiNiFi, Kafka, Kafka Connect,
Kafka Streams, Flink, Flink SQL, SMM, SRM, SR and EFM.
Previously at E&Y, HPE, Pivotal & Hortonworks

© 2021 Cloudera, Inc. All rights reserved. 4
Cloudera Communities
Got questions? Leverage community.cloudera.com
Join our meetup:
www.meetup/pro/futureofdata

© 2021 Cloudera, Inc. All rights reserved. 5
Data
Sources
Analysts
Engineers
Scientists
Developers
Data
Users
Real-time
Batch
Structured
Unstructured
Data Lifecycle
integration for better
user productivity and
faster time to value
Hybrid & Multi-Cloud
to leverage existing
investments and
reduce risk
Secure & Governed
to simplify data
protection, sharing
and compliance
Open & Extensible
to support more
use cases faster
and at lower cost
A HYBRID / MULTI-CLOUD DATA PLATFORM AND
AN INTEGRATED SUITE OF SECURE ANALYTIC APPS
Cloudera Data Platform
Data
Visualization
Machine
Learning
Data
Warehouse
Virtual
Data Lake
Data
Engineering
Data
Collection
Streaming
Analytics
Operational
Database

© 2021 Cloudera, Inc. All rights reserved. 6
Where? Cloudera Public Cloud
CDP services are optimized for the elastic
compute & ‘always-on’ storage services provided
by any cloud provider
Web service hosted and managed by Cloudera
Hosted in the your cloud environment, but
managed by the CDP Management Console
Shared Data Experience (SDX) technologies form
a secure and governed data lake backed by object
storage (S3, ADLS, GCS)
Flow Management Streams Messaging Streaming Analytics
AWS AZURE GOOGLE

© 2021 Cloudera, Inc. All rights reserved. 7
What use cases would you like to see for the next demo jam?
10% 20% 30% 40% 50%
IoT Edge to Cloud 47%
Detect Event Streams 17%
Optimize Data Streams 36%

© 2021 Cloudera, Inc. All rights reserved. 8
IoT Edge to Cloud
Kafka Topics
Voltage Temp
Energy Usage
Streaming
Analytics
Alerting
Conditions

© 2021 Cloudera, Inc. All rights reserved. 9
DISTRIBUTE DATA COLLECTION AND PROCESSING ACROSS PLATFORMS
CDP
Cloudera Edge Management and Cloudera Flow Management
IoT
Application
Energy Logs
Kafka Topics
Voltage Temp
SQL Stream Builder
IoT Edge

Question #1
- What is the most diﬃcult part of an Edge Flow?
- Gateway Agent
- Edge Data Collection
- Processing Data

Question #2
- What do you want to see next?
- Deliver the data to a data store
- Deliver to Kafka topic

Question #3
- Do you want to see real-time decision making on
streams?
- Yes
- No

Question #4
- What data store should I send my data to?
- Real-time Data Mart (Such as Apache Impala and Apache
Kudu)
- Object Storage (Such as HDFS on S3, ADLSv2, GCS,
Apache Ozone)
- Operational Database (Such as Apache HBase)
- In Stream Only (Apache Kafka with Data Window in SQL
Stream Builder)
- None
- All

© 2021 Cloudera, Inc. All rights reserved. 15
TH N Y U

Contenu connexe

Tendances

Data science online camp using the flipn stack for edge ai (flink, nifi, pu...

Timothy Spann

ApacheCon 2021: Cracking the nut with Apache Pulsar (FLiP) by Timothy Spann Wednesday 17:10 UTC - Cracking the Nut, Solving Edge AI with Apache Tools and Frameworks Wednesday 17:10 UTC Cracking the Nut, Solving Edge AI with Apache Tools and Frameworks Today, data is being generated from devices and containers living at the edge of networks, clouds and data centers. We need to run business logic, analytics and deep learning at the edge before we start our real-time streaming flows. Fortunately using the all Apache FLiP Stack we can do this with ease! Streaming AI Powered Analytics From the Edge to the Data Center is now a simple use case. With MiNiFi we can ingest the data, do data checks, cleansing, run machine learning and deep learning models and route our data in real-time to Apache NiFi and Apache Pulsar for further transformations and processing. Apache Flink will provide our advanced streaming capabilities fed real-time via Apache Kafka topics. Apache MXNet models will run both at the edge and in our data centers via Apache NiFi and MiNiFi. Our final data will be stored in various Apache datastores. Event-Driven Microservices in Apache Pulsar Functions. Tools: Apache Flink, Apache Pulsar, Apache NiFi, MiNiFi, Apache MXNet

ApacheCon 2021: Cracking the nut with Apache Pulsar (FLiP)

Timothy Spann

Python web conference 2022 apache pulsar development 101 with python (f li-...

Timothy Spann

PortoTechHub - Hail Hydrate! From Stream to Lake with Apache Pulsar and Friends https://portotechhub.com/conference-2021/ Timothy Spann Developer Advocate StreamNative A cloud data lake that is empty is not useful to anyone. How can you quickly, scalably and reliably fill your cloud data lake with diverse sources of data you already have and new ones you never imagined you needed. Utilizing open source tools from Apache, the FLiP stack enables any data engineer, programmer or analyst to build reusable modules with low or no code. FLiP utilizes Apache NiFi, Apache Pulsar, Apache Flink and MiNiFi agents to load CDC, Logs, REST, XML, Images, PDFs, Documents, Text, semistructured data, unstructured data, structured data and a hundred data sources you could never dream of streaming before. I will teach you how to fish in the deep end of the lake and return a data engineering hero. Let's hope everyone is ready to go from 0 to Petabyte hero. TRACK RIBEIRA Fri 07:00 — 50 min 19-Nov-2021

PortoTechHub - Hail Hydrate! From Stream to Lake with Apache Pulsar and Friends

Timothy Spann

Automation + dev ops summit hail hydrate! from stream to lake

Timothy Spann

Cracking the nut, solving edge ai with apache tools and frameworks

Timothy Spann

https://altinity.com/osa-con-2021/ An empty real-time SQL data warehouse is not useful to anyone. How can you load data quickly from diverse data sources? Utilizing open source tools from Apache, the FLiP stack enables any data engineer, programmer or analyst to build reusable modules with low or no code. We’ll show how to use them to load CDC, logs, events, XML, images, and many other types of data into ClickHouse and similar data warehouses. Osacon 2021 hello hydrate! from stream to clickhouse with apache pulsar and friends

Osacon 2021 hello hydrate! from stream to clickhouse with apache pulsar and...

Timothy Spann

Big mountain data and dev conference apache pulsar with mqtt for edge compu...

Timothy Spann

Using the FLaNK Stack for edge ai (flink, nifi, kafka, kudu) ntroducing the FLaNK stack which combines Apache Flink, Apache NiFi, Apache Kafka and Apache Kudu to build fast applications for IoT, AI, rapid ingest. FLaNK provides a quick set of tools to build applications at any scale for any streaming and IoT use cases. https://www.flankstack.dev/ Tools Apache Flink, Apache Kafka, Apache NiFi, MiNiFi, Apache MXNet, Apache Kudu, Apache Impala, Apache HDFS References https://www.datainmotion.dev/2019/08/rapid-iot-development-with-cloudera.html https://www.datainmotion.dev/2019/09/powering-edge-ai-for-sensor-reading.html https://www.datainmotion.dev/2019/05/dataworks-summit-dc-2019-report.html https://www.datainmotion.dev/2019/03/using-raspberry-pi-3b-with-apache-nifi.html Track Community and Industry Impact

Using the FLaNK Stack for edge ai (flink, nifi, kafka, kudu)

Timothy Spann

Using FLiP with influxdb for edgeai iot at scale 2022

Timothy Spann

Using the flipn stack for edge ai (flink, nifi, pulsar)

Timothy Spann

Let's build a simple ingest to cloud datawarehouse with low code

Timothy Spann

Using the FLiPN stack for edge ai (flink, nifi, pulsar)

Timothy Spann

(VIRTUAL) Hail Hydrate! From Stream to Lake Using Open Source - Timothy J Spann, StreamNative https://osselc21.sched.com/event/lAPi?iframe=no A cloud data lake that is empty is not useful to anyone. How can you quickly, scalably and reliably fill your cloud data lake with diverse sources of data you already have and new ones you never imagined you needed. Utilizing open source tools from Apache, the FLiP stack enables any data engineer, programmer or analyst to build reusable modules with low or no code. FLiP utilizes Apache NiFi, Apache Pulsar, Apache Flink and MiNiFi agents to load CDC, Logs, REST, XML, Images, PDFs, Documents, Text, semistructured data, unstructured data, structured data and a hundred data sources you could never dream of streaming before. I will teach you how to fish in the deep end of the lake and return a data engineering hero. Let's hope everyone is ready to go from 0 to Petabyte hero. https://osselc21.sched.com/event/lAPi/virtual-hail-hydrate-from-stream-to-lake-using-open-source-timothy-j-spann-streamnative

Hail hydrate! from stream to lake using open source

Timothy Spann

Using the FLaNK Stack for edge ai (apache mxnet, apache flink, apache nifi, a...

Timothy Spann

Cloud lunch and learn real-time streaming in azure

Timothy Spann

Incrementally streaming rdbms data to your data lake automagically

Timothy Spann

Speaker: Cedrick Lunven, Developer Advocate, DataStax Speaker Bio: Cedrick is a Developer Advocate at DataStax where he finds opportunities to share his passions by speaking about developing distributed architectures and implementing reference applications for developers. In 2013, he created FF4j, an open source framework for Feature Toggle which he still actively maintains. He is now contributor in JHipster team. Talk Synopsis: We have all introduced more or less functional programming and asynchronous operations into our applications in order to speed up and distribute treatments (e.g., multi-threading, future, completableFuture, etc.). To build truly non-blocking components, optimize resource usage, and avoid "callback hell" you have to think reactive—everything is an event. From the frontend UI to database communications, it’s now possible to develop Java applications as fully reactive with frameworks like Spring WebFlux and Reactor. With high throughput and tunable consistency, applications built on top of Apache Cassandra™ fit perfectly within this pattern. DataStax has been developing Apache Cassandra drivers for years, and in the latest version of the enterprise driver we introduced reactive programming. During this session we will migrate, step by step, a vanilla CRUD Java service (SpringBoot / SpringMVC) into reactive with both code review and live coding. Bring home a working project! Filmed at Skills Matter/Code Node London on 9th May 2019 as part of the Big Data LDN Meetup Blueprint Series. Meetup sponsored by DataStax.

Speed Up Your Apache Cassandra™ Applications: A Practical Guide to Reactive P...

Matt Stubbs

Learning the basics of Apache NiFi for iot OSS Europe 2020

Timothy Spann

ApacheCon 2021: Apache NiFi 101- introduction and best practices Thursday 14:10 UTC Apache NiFi 101: Introduction and Best Practices Timothy Spann In this talk, we will walk step by step through Apache NiFi from the first load to first application. I will include slides, articles and examples to take away as a Quick Start to utilizing Apache NiFi in your real-time dataflows. I will help you get up and running locally on your laptop, Docker DZone Zone Leader and Big Data MVB @PaasDev https://github.com/tspannhw https://www.datainmotion.dev/ https://github.com/tspannhw/SpeakerProfile https://dev.to/tspannhw https://sessionize.com/tspann/ https://www.slideshare.net/bunkertor

ApacheCon 2021: Apache NiFi 101- introduction and best practices

Timothy Spann

Tendances (20)

Data science online camp using the flipn stack for edge ai (flink, nifi, pu...

ApacheCon 2021: Cracking the nut with Apache Pulsar (FLiP)

Python web conference 2022 apache pulsar development 101 with python (f li-...

PortoTechHub - Hail Hydrate! From Stream to Lake with Apache Pulsar and Friends

Automation + dev ops summit hail hydrate! from stream to lake

Cracking the nut, solving edge ai with apache tools and frameworks

Osacon 2021 hello hydrate! from stream to clickhouse with apache pulsar and...

Big mountain data and dev conference apache pulsar with mqtt for edge compu...

Using the FLaNK Stack for edge ai (flink, nifi, kafka, kudu)

Using FLiP with influxdb for edgeai iot at scale 2022

Using the flipn stack for edge ai (flink, nifi, pulsar)

Let's build a simple ingest to cloud datawarehouse with low code

Using the FLiPN stack for edge ai (flink, nifi, pulsar)

Hail hydrate! from stream to lake using open source

Using the FLaNK Stack for edge ai (apache mxnet, apache flink, apache nifi, a...

Cloud lunch and learn real-time streaming in azure

Incrementally streaming rdbms data to your data lake automagically

Speed Up Your Apache Cassandra™ Applications: A Practical Guide to Reactive P...

Learning the basics of Apache NiFi for iot OSS Europe 2020

ApacheCon 2021: Apache NiFi 101- introduction and best practices

Similaire à Live Demo Jam Expands: The Leading-Edge Streaming Data Platform with NiFi, Kafka, and Flink

Unconference Round Table Notes

Timothy Spann

Gen AI on Enterprise Cloud Apache NiFi Milvus Apache Kafka Apache Flink Cloudera Machine Learning Cloudera DataFlow https://medium.com/@tspann/building-a-milvus-connector-for-nifi-34372cb3c7fa https://www.meetup.com/futureofdata-princeton/events/300737266/ https://lu.ma/q7pcfyjn?source=post_page-----34372cb3c7fa--------------------------------&tk=TTyakY If you're interested in working with Generative AI on the cloud, this virtual workshop is for you. Tim Spann from Cloudera and Yujian Tang from Zilliz will cover how you can implement your own GenAI workflows on the cloud at enterprise scale. 9:00 - 9:05: Intro 9:05 - 9:15: What is Milvus 9:15 - 9:25: Cloudera Development Platform 9:25 - 10:00: Demo Location https://www.youtube.com/watch?v=IfWIzKsoHnA https://github.com/tspannhw/SpeakerProfile https://www.linkedin.com/in/yujiantang/

Generative AI on Enterprise Cloud with NiFi and Milvus

Timothy Spann

Meetup Streaming Data Pipeline Development 28 June 2023 6pm EST Milwaukee meetup https://www.meetup.com/futureofdata-princeton/events/292976004/ Details This will be a hybrid event with a Zoom. The in-person event will be in Milwaukee. In this interactive session, Tim will lead participants through how to best build streaming data pipelines. He will cover how to build applications from some common use cases and highlight tips, tricks, best practices and patterns. He will show how to build the easy way and then dive deep into the underlying open source technologies including Apache NiFi, Apache Flink, Apache Kafka and Apache Iceberg. If you wish to follow along, please download open source projects beforehand. You can also download this helpful streaming platform: https://docs.cloudera.com/csp-ce/latest/installation/topics/csp-ce-installing-ce.html All source code and slides will be shared for those interested in building their own FLaNK Apps. https://www.flankstack.dev/ https://www.thecapitalgrille.com/locations/wi/milwaukee/milwaukee/8027 The Capital Grille 310 W Wisconsin Ave, Milwaukee, WI 53203 limited seating, preference will be given to NLIT attendees A peak at the menu (Not Pizza) RISOTTO FRITTERS WITH FRESH MOZZARELLA AND PROSCIUTTO SLICED SIRLOIN WITH ROQUEFORT AND BALSAMIC ONIONS MINIATURE LOBSTER AND CRAB CAKES WILD MUSHROOM AND HERBED CHEESE You can join the meeting virtually here (no meat or cheese virtually):

Meetup Streaming Data Pipeline Development

Timothy Spann

Future of Data Milwaukee Meetup Streaming Data Pipeline Development 28 June 2023 Future of Data: New Jersey - Princeton, Edison, Holmdel This will be a hybrid event with a Zoom. The in-person event will be in Milwaukee. In this interactive session, Tim will lead participants through how to best build streaming data pipelines. He will cover how to build applications from some common use cases and highlight tips, tricks, best practices and patterns. He will show how to build the easy way and then dive deep into the underlying open source technologies including Apache NiFi, Apache Flink, Apache Kafka and Apache Iceberg. If you wish to follow along, please download open source projects beforehand. You can also download this helpful streaming platform: https://docs.cloudera.com/csp-ce/latest/installation/topics/csp-ce-installing-ce.html All source code and slides will be shared for those interested in building their own FLaNK Apps. https://www.flankstack.dev/ https://www.thecapitalgrille.com/locations/wi/milwaukee/milwaukee/8027 The Capital Grille 310 W Wisconsin Ave, Milwaukee, WI 53203 limited seating, preference will be given to NLIT attendees A peak at the menu (Not Pizza) RISOTTO FRITTERS WITH FRESH MOZZARELLA AND PROSCIUTTO SLICED SIRLOIN WITH ROQUEFORT AND BALSAMIC ONIONS MINIATURE LOBSTER AND CRAB CAKES WILD MUSHROOM AND HERBED CHEESE You can join the meeting virtually here (no meat or cheese virtually):

Future of Data Milwaukee Meetup Streaming Data Pipeline Development 28 June 2023

ssuser73434e

This is Part 4 of the GoldenGate series on Data Mesh - a series of webinars helping customers understand how to move off of old-fashioned monolithic data integration architecture and get ready for more agile, cost-effective, event-driven solutions. The Data Mesh is a kind of Data Fabric that emphasizes business-led data products running on event-driven streaming architectures, serverless, and microservices based platforms. These emerging solutions are essential for enterprises that run data-driven services on multi-cloud, multi-vendor ecosystems. Join this session to get a fresh look at Data Mesh; we'll start with core architecture principles (vendor agnostic) and transition into detailed examples of how Oracle's GoldenGate platform is providing capabilities today. We will discuss essential technical characteristics of a Data Mesh solution, and the benefits that business owners can expect by moving IT in this direction. For more background on Data Mesh, Part 1, 2, and 3 are on the GoldenGate YouTube channel: https://www.youtube.com/playlist?list=PLbqmhpwYrlZJ-583p3KQGDAd6038i1ywe Webinar Speaker: Jeff Pollock, VP Product (https://www.linkedin.com/in/jtpollock/) Mr. Pollock is an expert technology leader for data platforms, big data, data integration and governance. Jeff has been CTO at California startups and a senior exec at Fortune 100 tech vendors. He is currently Oracle VP of Products and Cloud Services for Data Replication, Streaming Data and Database Migrations. While at IBM, he was head of all Information Integration, Replication and Governance products, and previously Jeff was an independent architect for US Defense Department, VP of Technology at Cerebra and CTO of Modulant – he has been engineering artificial intelligence based data platforms since 2001. As a business consultant, Mr. Pollock was a Head Architect at Ernst & Young’s Center for Technology Enablement. Jeff is also the author of “Semantic Web for Dummies” and "Adaptive Information,” a frequent keynote at industry conferences, author for books and industry journals, formerly a contributing member of W3C and OASIS, and an engineering instructor with UC Berkeley’s Extension for object-oriented systems, software development process and enterprise architecture.

Data Mesh Part 4 Monolith to Mesh

Jeffrey T. Pollock

2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipelines https://www.meetup.com/futureofdata-newyork/events/298660453/ Unlocking Financial Data with Real-Time Pipelines (Flink Analytics on Stocks with SQL ) By Timothy Spann Financial institutions thrive on accurate and timely data to drive critical decision-making processes, risk assessments, and regulatory compliance. However, managing and processing vast amounts of financial data in real-time can be a daunting task. To overcome this challenge, modern data engineering solutions have emerged, combining powerful technologies like Apache Flink, Apache NiFi, Apache Kafka, and Iceberg to create efficient and reliable real-time data pipelines. In this talk, we will explore how this technology stack can unlock the full potential of financial data, enabling organizations to make data-driven decisions swiftly and with confidence. Introduction: Financial institutions operate in a fast-paced environment where real-time access to accurate and reliable data is crucial. Traditional batch processing falls short when it comes to handling rapidly changing financial markets and responding to customer demands promptly. In this talk, we will delve into the power of real-time data pipelines, utilizing the strengths of Apache Flink, Apache NiFi, Apache Kafka, and Iceberg, to unlock the potential of financial data. I will be utilizing NiFi 2.0 with Python and Vector Databases. Timothy Spann Principal Developer Advocate, Cloudera Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Kafka, Apache Pulsar, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science. https://twitter.com/PaaSDev https://www.linkedin.com/in/timothyspann/ https://medium.com/@tspann https://github.com/tspannhw/FLiPStackWeekly/

2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...

Timothy Spann

GSJUG: Mastering Data Streaming Pipelines 09May2023 https://www.meetup.com/futureofdata-princeton/events/293233881/ This is a repost from the Garden State Java Users Group Event. Join me at https://www.meetup.com/garden-state-java-user-group/events/293229660/ See: https://www.eventbrite.com/e/mastering-data-streaming-pipelines-tickets-627677218457?_ga=2.253257801.1787151623.1682868226-741104479.1678110925 Please note that registration via EventBrite is required to attend either in-person or online. We are happy to announce that Tim Spann will be our special guest for the May 9, 2023 meeting! Abstract: In this session, Tim will show you some best practices that he has discovered over the last seven years in building data streaming applications including IoT, CDC, Logs, and more. In his modern approach, we utilize several Apache frameworks to maximize the best features of all. We often start with Apache NiFi as the orchestrator of streams flowing into Apache Kafka. From there we build streaming ETL with Apache Flink, enhance events with NiFi enrichment. We build continuous queries against our topics with Flink SQL. We will show where Java fits in as sources, enrichments, NiFi processors and sinks. We hope to see you on May 9! Speaker Timothy Spann Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Pulsar, Apache Kafka, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science. In this session, Tim will show you some best practices that he has discovered over the last seven years in building data streaming applications, including IoT, CDC, Logs, and more. In his modern approach, we utilize several Apache frameworks to maximize the best features of all. We often start with Apache NiFi as the orchestrator of streams flowing into Apache Kafka. From there, we build streaming ETL with Apache Flink, enhance events with NiFi enrichment. We build continuous queries against our topics with Flink SQL. We will show where Java fits in as sources, enrichments, NiFi processors, and sinks. https://www.eventbrite.com/e/mastering-data-streaming-pipelines-tickets-627677218457?_ga=2.253257801.178

GSJUG: Mastering Data Streaming Pipelines 09May2023

Timothy Spann

April 2024 - NLIT Cloudera Real-Time LLM Streaming 2024

Timothy Spann

The data that organizations are required to analyze in order to make informed decisions is growing at an unprecedented rate. Companies have to capture the window of opportunity and become not just data driven, but event driven. In this talk, we will talk around addressing these issues and look into ways to bridge the on-premise kafka deployments with GCP stack for different use cases and personas. This will be followed by architecture examples on How do you deploy kafka and integrate with the rest of the GCP stack.

Hybrid Streaming Analytics for Apache Kafka Users | Firat Tekiner, Google

HostedbyConfluent

The data that organizations are required to analyze in order to make informed decisions is growing at an unprecedented rate. Companies have to capture the window of opportunity and become not just data driven, but event driven. In this talk, we will talk around addressing these issues and look into ways to bridge the on-premise kafka deployments with GCP stack for different usecases and personas. This will be followed by architecture examples on How do you deploy kafka and integrate with the rest of the GCP stack.

Hybrid Streaming Analytics for Apache Kafka Users | Firat Tekiner, Google

HostedbyConfluent

Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines https://www.youtube.com/watch?v=Yeua8NlzQ3Y https://www.conf42.com/Large_Language_Models_LLMs_2024_Tim_Spann_generative_ai_streaming Adding Generative AI to Real-Time Streaming Pipelines Abstract Let’s build streaming pipelines that convert streaming events into prompts, call LLMs, and process the results. Summary Tim Spann: My talk is adding generative AI to real time streaming pipelines. I'm going to discuss a couple of different open source technologies. We'll touch on Kafka, Nifi, Flink, Python, Iceberg. All the slides, all the code and GitHub are out there. Llm, if you didn't know, is rapidly evolving. There's a lot of different ways to interact with models. That enrichment, transformation, processing really needs tools. The amount of models and projects and software that are available is massive. Nifi supports hundreds of different inputs and can convert them on the fly. Great way to distribute your data quickly to whoever needs it without duplication, without tight coupling. Fun to find new things to integrate into. So what we can do is, well, I want to get a meetup chat going. I have a processor here that just listens for events as they come from slack. And then I'm going to clean it up, add a couple fields and push that out to slack. Every model is a little bit of different tweaking. Nifi acts as a whole website. And as you see here, it can be get, post, put, whatever you want. We send that response back to flink and it shows up here. Thank you for attending this talk. I'm going to be speaking at some other events very shortly. Transcript This transcript was autogenerated. To make changes, submit a PR. Hi, Tim Spann here. My talk is adding generative AI to real time streaming pipelines, and we're here for the large language model conference at Comp 42, which is always a nice one, great place to be. I'm going to discuss a couple of different open source technologies that work together to enable you to build real time pipelines using large language models. So we'll touch on Kafka, Nifi, Flink, Python, Iceberg, and I'll show you a little bit of each one in the demos. I've been working with data machine learning, streaming IoT, some other things for a number of years, and you could contact me at any of these places, whether Twitter or whatever it's called, some different blogs, or in person at my meetups and at different conferences around the world. I do a weekly newsletter, cover streaming ML, a lot of LLM, open source, Python, Java, all kinds of fun stuff, as I mentioned, do a bunch of different meetups. They are not just in the east coast of the US, they are available virtually live, and I also put them on YouTube, and if you need them somewhere else, let me know. We publish all the slides, all the code and GitHub. Everything you need is out there. Let's get into the talk. Llm, if you didn't know, is rapidly evolving. While you're typing down the things that you use, it

Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines

Timothy Spann

PartnerSkillUp_Enable a Streaming CDC Solution Tim Spann Principal Developer Advocate in Data In Motion for Cloudera, Global https://attend.cloudera.com/skillupseriesseptember14 Streaming Change Data Capture (CDC) Two Unique Ways In this next session, learn how to use Debezium with Flink, Kafka, and NiFi for Change Data Capture using two different mechanisms: Kafka Connect and Flink SQL. With the virtual nature of today's world, streaming data is more critical than ever. Join Cloudera Chief Data-In-Motion Principal, Tim Spann, and Partner Solution Engineer, Salvador Alamazan as they look closely at key CDC use cases, discuss why Debezium is the best option for handling CDC and use examples to show you how to demonstrate value. This is a must-attend experience! Zoom Webinar September 14, 2023 10:00am–11:00am EDT FLaNK Stack Apache NiFi Apache Flink Apache Kafka Kafka Connect Flink SQL Cloudera DataFlow Cloudera SQL Stream Builder Cloudera Streams Messages Manager Debezium Postgresql IBM DB2 Oracle DB

PartnerSkillUp_Enable a Streaming CDC Solution

Timothy Spann

Part 2: A Visual Dive into Machine Learning and Deep Learning  

Cloudera, Inc.

Streaming Data and Stream Processing with Apache Kafka

confluent

Continus sql with sql stream builder

Timothy Spann

"Unlike just a few years ago, today the lakehouse architecture is an established data platform embraced by all major cloud data companies such as AWS, Azure, Google, Oracle, Microsoft, Snowflake and Databricks. This session kicks off with a technical, no-nonsense introduction to the lakehouse concept, dives deep into the lakehouse architecture and recaps how a data lakehouse is built from the ground up with streaming as a first-class citizen. Then we focus on serverless for streaming use cases. Serverless concepts are well-known from developers triggering hundreds of thousands of AWS Lambda functions at a negligible cost. However, the same concept becomes more interesting when looking at data platforms. We have all heard about the principle ""It runs best on Powerpoint"", so I decided to skip slides here and bring a serverless demo instead: A hands-on, fun, and interactive serverless streaming use case example where we ingest live events from hundreds of mobile devices (don't miss out - bring your phone and be part of it!!). Based on this use case I will critically explore how much of a modern lakehouse is serverless and how we implemented that at Databricks (spoiler alert: serverless is everywhere from data pipelines, workflows, optimized Spark APIs, to ML). TL;DR benefits for the Data Practitioners: -Recap the OSS foundation of the Lakehouse architecture and understand its appeal - Understand the benefits of leveraging a lakehouse for streaming and what's there beyond Spark Structured Streaming. - Meat of the talk: The Serverless Lakehouse. I give you the tech bits beyond the hype. How does a serverless lakehouse differ from other serverless offers? - Live, hands-on, interactive demo to explore serverless data engineering data end-to-end. For each step we have a critical look and I explain what it means, e.g for you saving costs and removing operational overhead."

Standing on the Shoulders of Open-Source Giants: The Serverless Realtime Lake...

HostedbyConfluent

Customer migration to Azure SQL database, December 2019

George Walters

SentinelOne-Connector-For-Fortinet-Launch-Deck-Final.pptx Optimizing slideshow... SentinelOne-Connector-For-Fortinet-Launch-Deck-Final.pptx Optimizing slideshow... SentinelOne-Connector-For-Fortinet-Launch-Deck-Final.pptx Optimizing slideshow... SentinelOne-Connector-For-Fortinet-Launch-Deck-Final.pptx Optimizing slideshow... SentinelOne-Connector-For-Fortinet-Launch-Deck-Final.pptx Optimizing slideshow... SentinelOne-Connector-For-Fortinet-Launch-Deck-Final.pptx Optimizing slideshow... the advanced robot went over the hill to find the bunker filled with explosives

Slides-Discover-Power-of-Live-Data(2).pdf

butthead7

Introduction to Apache NiFi 1.10 Parameters, Stateless, RetryFlowFile, Backpressure prediction, parquetreader, parquetwriter, postslack, remote inputport in process group. Dec 2019, Timothy Spann, Field Engineer, Data in Motion Princeton Meetup 10-dec-2019 https://www.meetup.com/futureofdata-princeton/events/266496424/ Hosted By PGA Fund at: https://pga.fund/coworking-space/ Princeton Growth Accelerator 5 Independence Way, 4th Floor, Princeton, NJ

Introduction to Apache NiFi 1.10

Timothy Spann

Delivering Data Democratization in the Cloud with Snowflake

Kent Graziano

Similaire à Live Demo Jam Expands: The Leading-Edge Streaming Data Platform with NiFi, Kafka, and Flink (20)

Unconference Round Table Notes

Generative AI on Enterprise Cloud with NiFi and Milvus

Meetup Streaming Data Pipeline Development

Future of Data Milwaukee Meetup Streaming Data Pipeline Development 28 June 2023

Data Mesh Part 4 Monolith to Mesh

2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...

GSJUG: Mastering Data Streaming Pipelines 09May2023

April 2024 - NLIT Cloudera Real-Time LLM Streaming 2024

Hybrid Streaming Analytics for Apache Kafka Users | Firat Tekiner, Google

Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines

PartnerSkillUp_Enable a Streaming CDC Solution

Part 2: A Visual Dive into Machine Learning and Deep Learning  

Streaming Data and Stream Processing with Apache Kafka

Continus sql with sql stream builder

Standing on the Shoulders of Open-Source Giants: The Serverless Realtime Lake...

Customer migration to Azure SQL database, December 2019

Slides-Discover-Power-of-Live-Data(2).pdf

Introduction to Apache NiFi 1.10

Delivering Data Democratization in the Cloud with Snowflake

Plus de Timothy Spann

Building Real-Time Pipelines With FLaNK Timothy Spann, Principal Developer Advocate, Streaming - Cloudera Future of Data meetup, startup grind, AI Camp The combination of Apache Flink, Apache NiFi, and Apache Kafka for building real-time data processing pipelines is extremely powerful, as demonstrated by this case study using the FLaNK-MTA project. The project leverages these technologies to process and analyze real-time data from the New York City Metropolitan Transportation Authority (MTA). FLaNK-MTA demonstrates how to efficiently collect, transform, and analyze high-volume data streams, enabling timely insights and decision-making. Apache NiFi Apache Kafka Apache Flink Apache Iceberg LLM Generative AI Slack Postgresql

DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK

Timothy Spann

Real-Time AI Streaming - AI Max Princeton

Timothy Spann

2024 XTREMEJ_ Building Real-time Pipelines with FLaNK_ A Case Study with Tra...

Timothy Spann

28March2024-Codeless-Generative-AI-Pipelines https://www.meetup.com/futureofdata-princeton/events/299440871/ https://www.meetup.com/real-time-analytics-meetup-ny/events/299290822/ ******Note***** The event is seat-limited, therefore please complete your registration here. Only people completing the form will be able to attend. ----------------------- We're excited to invite you to join us in-person, for a Real-Time Analytics exploration! Join us for an evening of insights, networking as we delve into the OSS technologies shaping the field! Agenda: 05:30-06:00: Pizza and friends 06:00- 06:40: Codeless GenAI Pipelines with Flink, Kafka, NiFi 06:40- 07:20 Real-Time Analytics in the Corporate World: How Apache Pinot® Powers Industry Leaders 07:20-07:30 QNA Codeless GenAI Pipelines with Flink, Kafka, NiFi | Tim Spann, Cloudera Explore the power of real-time streaming with GenAI using Apache NiFi. Learn how NiFi simplifies data engineering workflows, allowing you to focus on creativity over technical complexities. I'll guide you through practical examples, showcasing NiFi's automation impact from ingestion to delivery. Whether you're a seasoned data engineer or new to GenAI, this talk offers valuable insights into optimizing workflows. Join us to unlock the potential of real-time streaming and witness how NiFi makes data engineering a breeze for GenAI applications! Real-Time Analytics in the Corporate World: How Apache Pinot® Powers Industry Leaders | Viktor Gamov, StarTree Explore how industry leaders like LinkedIn, Uber Eats, and Stripe are mastering real-time data with Viktor as your guide. Discover how Apache Pinot transforms data into actionable insights instantly. Viktor will showcase Pinot's features, including the Star-Tree Index, and explain why it's a game-changer in data strategy. This session is for everyone, from data geeks to business gurus, eager to uncover the future of tech. Join us and be wowed by the power of real-time analytics with Apache Pinot! ------- Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Kafka, Apache Pulsar, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more.

28March2024-Codeless-Generative-AI-Pipelines

Timothy Spann

https://princetonacm.acm.org/tcfpro/ 18th Annual IEEE IT Professional Conference (ITPC) Armstrong Hall at The College of New Jersey Friday, March 15th, 2024 | 10:00 AM to 5:00 PM IT Professional Conference at Trenton Computer Festival IEEE Information Technology Professional Conference on Friday, March 15th, 2024 TCFPro24 Building Real-Time Generative AI Pipelines Building Real-Time Generative AI Pipelines In this talk, Tim will delve into the exciting realm of building real-time generative AI pipelines with streaming capabilities. The discussion will revolve around the integration of cutting-edge technologies to create dynamic and responsive systems that harness the power of generative algorithms. From leveraging streaming data sources to implementing advanced machine learning models, the presentation will explore the key components necessary for constructing a robust real-time generative AI pipeline. Practical insights, use cases, and best practices will be shared, offering a comprehensive guide for developers and data scientists aspiring to design and implement dynamic AI systems in a streaming environment. Tim will show a live demo showing we can use Apache NiFi to provide a live chat between a person in Slack and several LLM models all orchestrated with Apache NiFi, Apache Kafka and Python. We will use RAG against Chroma and Pinecone vector data stores, Hugging Face and WatsonX.AI LLM, and add additional context with NiFi lookups of stocks, weather and other data streams in real-time. Timothy Spann Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Pulsar, Apache Kafka, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science.

TCFPro24 Building Real-Time Generative AI Pipelines

Timothy Spann

2024 Build Generative AI for Non-Profits

Timothy Spann

Conf42-Python-Building Apache NiFi 2.0 Python Processors https://www.conf42.com/Python_2024_Tim_Spann_apache_nifi_2_processors Building Apache NiFi 2.0 Python Processors Abstract Let’s enhance real-time streaming pipelines with smart Python code. Adding code for vector databases and LLM. Summary Tim Spann: I'm going to be talking today, be building Apache 9520 Python processors. One of the main purposes of supporting Python in the streaming tool Apache Nifi is to interface with new machine learning and AI and Gen AI. He says Python is a real game changer for Cloudera. You're just going to add some metadata around it. It's a great way to pass a file along without changing it too substantially. We really need you to have Python 310 and again JDK 21 on your machine. You got to be smart about how you use these models. There are a ton of python processors available. You can use them in multiple ways. We're still in the early world of Python processors, so now's the time to start putting yours out there. Love to see a lot of people write their own. When we are parsing documents here, again, this is the Python one I'm picking PDF. Lots of different things you could do. If you're interested on writing your own python code for Apache Nifi, definitely reach out and thank.

Conf42-Python-Building Apache NiFi 2.0 Python Processors

Timothy Spann

Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg with Stock Data and LLM Abstract In this talk, we’ll discuss how to use Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg to process and analyze stock data. We demonstrated the ingestion, processing, and analysis of stock data. Additionally, we illustrated how to use an LLM to generate predictions from the analyzed data. Karin Wolok Developer Relations, Dev Marketing, and Community Programming @ Project Elevate Karin Wolok's LinkedIn account Karin Wolok's twitter account Tim Spann Principal Developer Advocate @ Cloudera Tim Spann's LinkedIn account Tim Spann's twitter account https://www.conf42.com/Python_2024_Karin_Wolok_Tim_Spann_nifi__kafka_risingwave_iceberg_llm

Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...

Timothy Spann

2024 Feb AI Meetup NYC GenAI_LLMs_ML_Data Codeless Generative AI Pipelines https://www.aicamp.ai/event/eventdetails/W2024022214 apache nifi llm generative ai gen ai ml dl machine learning apache kafka apache flink postgresql python AI Meetup (NYC): GenAI, LLMs, ML and Data Feb 22, 05:30 PM EST Welcome to the monthly in-person AI meetup in New York City, in collaboration with Microsoft. Join us for deep dive tech talks on AI, GenAI, LLMs and machine learning, food/drink, networking with speakers and fellow developers Agenda: * 5:30pm~6:00pm: Checkin, Food/drink and networking * 6:00pm~6:10pm: Welcome/community update * 6:10pm~8:30pm: Tech talks * 8:30pm: Q&A, Open discussion Tech Talk: Searching and Reasoning Over Multimedia Data with Vector Databases and LMMs Speaker: Zain Hasan (Weaviate LinkedIn) Abstract: In this talk, Zain Hasan will discuss how we can use open-source multimodal embedding models in conjunction with large generative multimodal models that can that can see, hear, read, and feel data(!), to perform cross-modal search(searching audio with images, videos with text etc.) and multimodal retrieval augmented generation (MM-RAG) at the billion-object scale with the help of open source vector databases. I will also demonstrate, with live code demos, how being able to perform this cross-modal retrieval in real-time can enables users to use LLMs that can reason over their enterprise multimodal data. This talk will revolve around how we can scale the usage of multimodal embedding and generative models in production. Tech Talk: Codeless Generative AI Pipelines Speaker: Timothy Spann (Cloudera LinkedIn) Abstract: Join us for an insightful talk on leveraging the power of real-time streaming tools, specifically Apache NiFi, to revolutionize GenAI data engineering. In this session, we’ll explore how the integration of Apache NiFi can automate the entire process of prompt building, making it a seamless and efficient task. Speakers/Topics: Stay tuned as we are updating speakers and schedules. If you have a keen interest in speaking to our community, we invite you to submit topics for consideration: Submit Topics Sponsors: We are actively seeking sponsors to support our community. Whether it is by offering venue spaces, providing food/drink, or cash sponsorship. Sponsors will have the chance to speak at the meetups, receive prominent recognition, and gain exposure to our extensive membership base of 20,000+ local or 300K+ developers worldwide. Venue: Microsoft NYC - Times Square, 11 Times Square, New York, NY 10036 Room Name: Central Park West 6501 Community on Slack/Discord - Event chat: chat and connect with speakers and attendees - Sharing blogs, events, job openings, projects collaborations Join Slack (search and join the #newyork channel) | Join Discord

2024 Feb AI Meetup NYC GenAI_LLMs_ML_Data Codeless Generative AI Pipelines

Timothy Spann

DBA Fundamentals Group: Continuous SQL with Kafka and Flink 20-Feb-2024 In this talk, I will walk through how someone can set up and run continuous SQL queries against Kafka topics utilizing Apache Flink. We will walk through creating Kafka topics, schemas, and publishing data. We will then cover consuming Kafka data, joining Kafka topics, and inserting new events into Kafka topics as they arrive. This basic overview will show hands-on techniques, tips, and examples of how to do this. Tim Spann Tim Spann is the Principal Developer Advocate for Data in Motion @ Cloudera where he works with Apache Kafka, Apache Flink, Apache NiFi, Apache Iceberg, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal Field Engineer at Cloudera, a Senior Solutions Architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science.

DBA Fundamentals Group: Continuous SQL with Kafka and Flink

Timothy Spann

NY Open Source Data Meetup Feb 8 2024 Building Real-time Pipelines with FLaNK...

Timothy Spann

OSACon 2023_ Unlocking Financial Data with Real-Time Pipelines Unlocking Financial Data with Real-Time Pipelines Financial institutions thrive on accurate and timely data to drive critical decision-making processes, risk assessments, and regulatory compliance. However, managing and processing vast amounts of financial data in real-time can be a daunting task. To overcome this challenge, modern data engineering solutions have emerged, combining powerful technologies like Apache Flink, Apache NiFi, Apache Kafka, and Iceberg to create efficient and reliable real-time data pipelines. In this talk, we will explore how this technology stack can unlock the full potential of financial data, enabling organizations to make data-driven decisions swiftly and with confidence. Introduction: Financial institutions operate in a fast-paced environment where real-time access to accurate and reliable data is crucial. Traditional batch processing falls short when it comes to handling rapidly changing financial markets and responding to customer demands promptly. In this talk, we will delve into the power of real-time data pipelines, utilizing the strengths of Apache Flink, Apache NiFi, Apache Kafka, and Iceberg, to unlock the potential of financial data. Key Points to be Covered: Introduction to Real-Time Data Pipelines: a. The limitations of traditional batch processing in the financial domain. b. Understanding the need for real-time data processing. Apache Flink: Powering Real-Time Stream Processing: a. Overview of Apache Flink and its role in real-time stream processing. b. Use cases for Apache Flink in the financial industry. c. How Flink enables fast, scalable, and fault-tolerant processing of streaming financial data. Apache Kafka: Building Resilient Event Streaming Platforms: a. Introduction to Apache Kafka and its role as a distributed streaming platform. b. Kafka's capabilities in handling high-throughput, fault-tolerant, and real-time data streaming. c. Integration of Kafka with financial data sources and consumers. Apache NiFi: Data Ingestion and Flow Management: a. Overview of Apache NiFi and its role in data ingestion and flow management. b. Data integration and transformation capabilities of NiFi for financial data. c. Utilizing NiFi to collect and process financial data from diverse sources. Iceberg: Efficient Data Lake Management: a. Understanding Iceberg and its role in managing large-scale data lakes. b. Iceberg's schema evolution and table-level metadata capabilities. c. How Iceberg simplifies data lake management in financial institutions. Real-World Use Cases: a. Real-time fraud detection using Flink, Kafka, and NiFi. b. Portfolio risk analysis with Iceberg and Flink. c. Streamlined regulatory reporting leveraging all four technologies. Best Practices and Considerations: a. Architectural considerations when building real-time financial data pipelines. b. Ensuring data integrity, security, and compliance in real-time pipelines. c. Scalability an

OSACon 2023_ Unlocking Financial Data with Real-Time Pipelines

Timothy Spann

Building Real-time Travel Alerts In this session, we will walk through how to build a complete streaming application to send alerts based on travel advisories from public data. We will also join in other data sources of relevance and push out alerts. We will show you how to build this streaming application with Apache NiFi, Apache Kafka, and Apache Flink and show you when/why/how, and what to build to maximize performance, productivity, and ease of development. Let's get streaming. Apache Flink Apache Kafka Apache NiFi FLaNK Stack Tim Spann Big Data Conference Europe 2023

Building Real-Time Travel Alerts

Timothy Spann

JConWorld: Continuous SQL with Kafka and Flink In this talk, I will walk through how someone can setup and run continous SQL queries against Kafka topics utilizing Apache Flink. We will walk through creating Kafka topics, schemas and publishing data. We will then cover consuming Kafka data, joining Kafka topics and inserting new events into Kafka topics as they arrive. This basic over view will show hands-on techniques, tips and examples of how to do this. Tim Spann is the Principal Developer Advocate for Data in Motion @ Cloudera where he works with Apache Kafka, Apache Flink, Apache NiFi, Apache Iceberg, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal Field Engineer at Cloudera, a Senior Solutions Architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science. https://www.datainmotion.dev/p/about-me.html https://dzone.com/users/297029/bunkertor.html https://www.youtube.com/channel/UCDIDMDfje6jAvNE8DGkJ3_w?view_as=subscriber

JConWorld_ Continuous SQL with Kafka and Flink

Timothy Spann

[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines https://dssconf.pl/en/#agenda-section Integrating LLM with Streaming Data Pipelines Timothy Spann, Principal Developer Advocate, Cloudera APACHE NIFI, APACHE FLINK, APACHE KAFKA, LLM, HUGGINGFACE, REST, STREAMING LESS In this talk and demo I will walk through how to add LLMs to your streaming pipelines by integration through Apache NiFi. https://github.com/tspannhw/FLaNK-watsonx.ai Cloudera streaming llm generative ai slack to slack

[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines

Timothy Spann

Evolve 2023 NYC - Integrating AI Into Realtime Data Pipelines Demo

Timothy Spann

AIDevWorld 23 Apache NiFi 101 Introduction and Best Practices https://sched.co/1RoAO Timothy Spann, Cloudera, Principal Developer Advocate In this talk, we will walk step by step through Apache NiFi from the first load to first application. I will include slides, articles and examples to take away as a Quick Start to utilizing Apache NiFi in your real-time dataflows. I will help you get up and running locally on your laptop, Docker or in CDP Public Cloud. Wednesday November 1, 2023 12:00pm - 12:25pm PDT VIRTUAL AI DevWorld -- Main Stage https://app.hopin.com/events/api-world-2023-ai-devworld/stages Retail & E-Commerce AI (Industry AI Conference) Session Type OPEN TALK Track or Conference Retail & E-Commerce AI (Industry AI Conference), Industry AI Conference, VIRTUAL, Tensorflow & PyTorch & Open Source Frameworks (AI/ML Engineering Conference), AI/ML Engineering Conference, AI DevWorld In-Person/Virtual Virtual, Virtual Exclusive apache nifi Timothy Spann Cloudera Principal Developer Advocate for Data in Motion Tim Spann is the Principal Developer Advocate for Data in Motion @ Cloudera where he works with Apache Kafka, Apache Flink, Apache NiFi, Apache Iceberg, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal Field Engineer at Cloudera, a Senior Solutions Architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science. cloudera dataflow

AIDevWorldApacheNiFi101

Timothy Spann

26Oct2023_ Adding Generative AI to Real-Time Streaming Pipelines_ NYC Meetup.pdf ## Details **Important** Please complete your registration in this short form. For on-site we have limited room, so please confirm if you are attending in-person in Manhattan, NYC. -------------------------------------------------------------------------------------------- We're at StarTree, excited to join forces with our friends at Cloudera, for a meetup that is all about The Latest in Real-Time Analytics: Generative AI and LLM, featuring Apache Pinot and Apache NiFi. Join us for an insightful discussion about cutting-edge analytics, meet the community in person, and catch up over drinks and snacks. What's the plan ? 05:30-06:00 Pizza and Networking 06:00-06:35 Adding Generative AI to Real-Time Streaming Pipelines | Tim Spann, Principal Developer Advocate, Cloudera 06:35-07:10 Apache Pinot and Kafka an excellent pairing for refined palates | Tim Veil, VP of Solutions Engineering and Enablement, StarTree 07:10-07:20 QNA 07:20- 07:30 More Snacks and Networking ;) **Important** Seats are limited Please complete your registration in this short form. Adding Generative AI to Real-Time Streaming Pipelines | Timothy Spann In this talk, Tim will discuss the basics of real-time streaming, walk through the tools used including Apache NiFi, Apache Kafka and Apache Flink and show how to build a real-time streaming pipeline that sends prompts to LLMs hosted by the likes of Hugging Face, IBM and Cloudera. He will also discuss where real-time data stores like Apache Pinot come into play. He will show a detailed demonstration of a few use cases involving different sources of data including Kafka, Medium Articles and interactive Question and Response in Slack. He will then show you how you can build your own and where areas of growth exist. Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Kafka, Apache Pulsar, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. https://github.com/tspannhw/SpeakerProfile Apache Pinot and Kafka an excellent pairing for refined palates | Tim Veil The other Tim, Tim Veil, will dive into the history and architect

26Oct2023_Adding Generative AI to Real-Time Streaming Pipelines_ NYC Meetup

Timothy Spann

CoC23_ Looking at the New Features of Apache NiFi Apache NiFi has a lot of new features, processors and best practices that have arrived in the last year or so. I will walk through building flows using the latest tips, techniques and processor. I will and change a number of data flows utilizing the latest NiFi version and point out gotchas and some never dos. The deck will act as a take-away with notes, tips and guides to what we covered. ===> Any NiFi 1.23+ and 2.0 in progress features people want to see?

CoC23_ Looking at the New Features of Apache NiFi

Timothy Spann

CoC23_ Let’s Monitor The Conditions at the Conference At home, I monitor the temperature, humidity, gas levels, ozone, air quality, and other features around my desk.Let’s bring this to the different spots around the conference including lunch tables, vendor booths, hotel rooms, and more. I need to know about these readings now, not when I get back home from the conference. We need to get these sensor readings immediately in case we need to turn on a fan or move to another area. We will also see if my talk produces a lot of hot air!?!??My setup is pretty simple, a raspberry pi, a breakout garden sensor mount, and as many sensors as I can carry. All code and parameters will be available ASF-licensed in github.The software stack is Python and Java, MQTT, HTML, JQuery, and Apache NiFi, Apache Kafka and Apache Flink.

CoC23_ Let’s Monitor The Conditions at the Conference

Timothy Spann

Plus de Timothy Spann (20)

DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK

Real-Time AI Streaming - AI Max Princeton

2024 XTREMEJ_ Building Real-time Pipelines with FLaNK_ A Case Study with Tra...

28March2024-Codeless-Generative-AI-Pipelines

TCFPro24 Building Real-Time Generative AI Pipelines

2024 Build Generative AI for Non-Profits

Conf42-Python-Building Apache NiFi 2.0 Python Processors

Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...

2024 Feb AI Meetup NYC GenAI_LLMs_ML_Data Codeless Generative AI Pipelines

DBA Fundamentals Group: Continuous SQL with Kafka and Flink

NY Open Source Data Meetup Feb 8 2024 Building Real-time Pipelines with FLaNK...

OSACon 2023_ Unlocking Financial Data with Real-Time Pipelines

Building Real-Time Travel Alerts

JConWorld_ Continuous SQL with Kafka and Flink

[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines

Evolve 2023 NYC - Integrating AI Into Realtime Data Pipelines Demo

AIDevWorldApacheNiFi101

26Oct2023_Adding Generative AI to Real-Time Streaming Pipelines_ NYC Meetup

CoC23_ Looking at the New Features of Apache NiFi

CoC23_ Let’s Monitor The Conditions at the Conference

Dernier

In an era where security concerns are paramount, the integration of artificial intelligence (AI) into CCTV cameras has revolutionized surveillance capabilities. One of the most significant advancements is the ability to achieve real-time threat detection, enabling immediate responses to potential security breaches. This blog explores how AI is reshaping surveillance through real-time threat detection and the implications of this technology.

Optimizing AI for immediate response in Smart CCTV

shikhaohhpro

Azure Native Qumulo scales elastically for common High Performance Compute (HPC) workloads based on application requirements for: Financial Services, Automotive, Genomics / Life Sciences, Media and Entertainment, Energy, Oil and Gas, etc. Performance can be dialed UP (and back down) much higher than the examples shown here. These slides offer a glimpse into ANQ's HPC capabilities, although at a smaller scale. We invite YOU to do your own testing (with a free ANQ trial) and work with us to test your HPC workloads in Azure.

Azure_Native_Qumulo_High_Performance_Compute_Benchmarks.pdf

ryanfarris8

+971565801893 Mtp-Kit (500MG) Prices » Dubai [(+971565801893**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Leen Whatsapp +971565801893 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971565801893''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971565801893' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Clinic in Abu Dhabi, United Arab Emirates.+971565801893

+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...

Health

MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...

Jittipong Loespradit

Learn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdf

kalichargn70th171

Craft an AI & Machine Learning Pitch with our Editable Professional PowerPoint Template. Ignite your AI & Machine Learning pitch with our cutting-edge PowerPoint template tailored for the industry. Perfect for AI conferences, investor presentations, sales pitches to tech-focused companies, training sessions, and educational programs. - 20+ editable slides: Get a variety of options to choose from for your presentation. - Time-saving solution: Download, replace text/images with a few clicks. - User-friendly customization: Easy to use and personalize. - Modern and attractive design: Captivating visuals, sleek layout. - Tailored to your requirements: Fully alterable for customization. - Well-organized slides: Complete control over content. - Thematic specificity: Reflects healthcare industry with relevant graphics. - Showcase your business idea: Communicate value proposition effectively.

AI & Machine Learning Presentation Template

Presentation.STUDIO

%in ivory park+277-882-255-28 abortion pills for sale in ivory park

masabamasaba

Data spaces in distributed environments should be allowed to evolve in agile ways providing data space owners with large flexibility about which data they store. Agility and heterogeneity, however, jeopardize data exchanges because representations may build on varying ontologies and data consumers may not rely on the semantic correctness of their queries in the context of semantically heterogeneous, evolving data spaces. Graph data spaces are one example of a powerful model for representing and querying data whose semantics may change over time. To assert and enforce conditions on individual graph data spaces, shape languages (e.g SHACL) have been developed. We investigate the question of how querying and programming can be guarded by reasoning over SHACL constraints in a distributed setting and we sketch a picture of how a future landscape based on semantically heterogeneous data spaces might look like.

Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...

Steffen Staab

10 Trends Likely to Shape Enterprise Technology in 2024

Mind IT Systems

%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein

masabamasaba

Define the academic and professional writing..pdf

PearlKirahMaeRagusta1

%in Midrand+277-882-255-28 abortion pills for sale in midrand

masabamasaba

Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...

SelfMade bd

A Secure and Reliable Document Management System is Essential.docx

ComplianceQuest1

%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein

masabamasaba

The title is not connected to what is inside

shinachiaurasa2

MakeMyPass" Online Bus Pass Management System illustrates the flow of activities and actions that occur within the system to accomplish specific tasks or use cases. This type of diagram focuses on representing the sequence of activities and decision points involved in a particular process. Below is an example outline and description of key elements that could be included in an Activity Diagram for the system:

BUS PASS MANGEMENT SYSTEM USING PHP.pptx

alwaysnagaraju26

%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain

masabamasaba

Software Quality Assurance Interview Questions

Arshad QA

At the recent Microsoft Ignite 2023 conference, Microsoft unveiled a groundbreaking strategy that will redefine enterprise work management. The plan involves integrating Microsoft’s key planning tools, Microsoft To Do, Microsoft Planner, and Microsoft Project for the web into a unified experience called “Microsoft Planner.” What does this new strategy from Microsoft mean for current users? Join us and learn how best to take advantage of this announcement while gaining a clear path on how to elevate the current state of Microsoft Planner from a basic task manager to a comprehensive tool for Enterprise Work Management using OnePlan. Learn how OnePlan’s integration with Microsoft Planner allows for strategic alignment with business goals through advanced features like strategic planning, portfolio management, resource management, financial management, and more!

Introducing Microsoft’s new Enterprise Work Management (EWM) Solution

OnePlan Solutions

Dernier (20)

Optimizing AI for immediate response in Smart CCTV

Azure_Native_Qumulo_High_Performance_Compute_Benchmarks.pdf

+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...

MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...

Learn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdf

AI & Machine Learning Presentation Template

%in ivory park+277-882-255-28 abortion pills for sale in ivory park

Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...

10 Trends Likely to Shape Enterprise Technology in 2024

%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein

Define the academic and professional writing..pdf

%in Midrand+277-882-255-28 abortion pills for sale in midrand

Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...

A Secure and Reliable Document Management System is Essential.docx

%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein

The title is not connected to what is inside

BUS PASS MANGEMENT SYSTEM USING PHP.pptx

%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain

Software Quality Assurance Interview Questions

Introducing Microsoft’s new Enterprise Work Management (EWM) Solution

Live Demo Jam Expands: The Leading-Edge Streaming Data Platform with NiFi, Kafka, and Flink

1. Live Demo Jam Expands: The Leading-Edge Streaming Data Platform with NiFi, Kafka, and Flink Live demo by Tim Spann, Principal DataFlow Field Engineer

2. © 2021 Cloudera, Inc. All rights reserved. 2 Housekeeping items for today • Ask your questions via the Q&A widget • Select “YES” to the exit survey to be contact by Cloudera • Check out additional resources to learn more

3. © 2021 Cloudera, Inc. All rights reserved. 3 Who? Timothy Spann Twitter - @PaasDev // Blog: www.datainmotion.dev Frequent speaker at major conferences and events. Principal DataFlow Field Engineer for streaming around Apache NiFi, NiFi Registry, MiNiFi, Kafka, Kafka Connect, Kafka Streams, Flink, Flink SQL, SMM, SRM, SR and EFM. Previously at E&Y, HPE, Pivotal & Hortonworks

5. © 2021 Cloudera, Inc. All rights reserved. 5 Data Sources Analysts Engineers Scientists Developers Data Users Real-time Batch Structured Unstructured Data Lifecycle integration for better user productivity and faster time to value Hybrid & Multi-Cloud to leverage existing investments and reduce risk Secure & Governed to simplify data protection, sharing and compliance Open & Extensible to support more use cases faster and at lower cost A HYBRID / MULTI-CLOUD DATA PLATFORM AND AN INTEGRATED SUITE OF SECURE ANALYTIC APPS Cloudera Data Platform Data Visualization Machine Learning Data Warehouse Virtual Data Lake Data Engineering Data Collection Streaming Analytics Operational Database

6. © 2021 Cloudera, Inc. All rights reserved. 6 Where? Cloudera Public Cloud CDP services are optimized for the elastic compute & ‘always-on’ storage services provided by any cloud provider Web service hosted and managed by Cloudera Hosted in the your cloud environment, but managed by the CDP Management Console Shared Data Experience (SDX) technologies form a secure and governed data lake backed by object storage (S3, ADLS, GCS) Flow Management Streams Messaging Streaming Analytics AWS AZURE GOOGLE

9. © 2021 Cloudera, Inc. All rights reserved. 9 DISTRIBUTE DATA COLLECTION AND PROCESSING ACROSS PLATFORMS CDP Cloudera Edge Management and Cloudera Flow Management IoT Application Energy Logs Kafka Topics Voltage Temp SQL Stream Builder IoT Edge

10. Question #1 - What is the most diﬃcult part of an Edge Flow? - Gateway Agent - Edge Data Collection - Processing Data

11. Question #2 - What do you want to see next? - Deliver the data to a data store - Deliver to Kafka topic

12. Question #3 - Do you want to see real-time decision making on streams? - Yes - No

13. Question #4 - What data store should I send my data to? - Real-time Data Mart (Such as Apache Impala and Apache Kudu) - Object Storage (Such as HDFS on S3, ADLSv2, GCS, Apache Ozone) - Operational Database (Such as Apache HBase) - In Stream Only (Apache Kafka with Data Window in SQL Stream Builder) - None - All

14. Q&A

Live Demo Jam Expands: The Leading-Edge Streaming Data Platform with NiFi, Kafka, and Flink

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Live Demo Jam Expands: The Leading-Edge Streaming Data Platform with NiFi, Kafka, and Flink

Similaire à Live Demo Jam Expands: The Leading-Edge Streaming Data Platform with NiFi, Kafka, and Flink (20)

Plus de Timothy Spann

Plus de Timothy Spann (20)

Dernier

Dernier (20)

Live Demo Jam Expands: The Leading-Edge Streaming Data Platform with NiFi, Kafka, and Flink