Confluent_AWS_ImmersionDay_Q42023.pdf

© 2022, Amazon Web Services, Inc. or its affiliates.
AWS + Confluent Immersion
Day
Building Secure, Event-Driven
Microservices with Confluent
Cloud on AWS
November 15, 2023

Today’s hosts:
2
Ahmed Zamzam
Senior Partner Solutions Architect,
Confluent
Nuno Barreto
Partner Sales Solutions Architect, AWS

Agenda
• Building Modern Streaming Analytics with Confluent on AWS
with Nuno Barreto, Partner Solutions Architect, AWS
• Event Streaming made easy with Confluent
with Ahmed Zamzam, Sr Partner Solutions Architect, Confluent
• Lab: Building end-to-end streaming data pipelines with Confluent Cloud
with Ahmed Zamzam, Sr Partner Solutions Architect, Confluent
3

Building Modern Streaming Analytics
with Confluent on AWS
Nuno Barreto (he/him)
Partner Sales Solutions Architect
Amazon Web Services (AWS)
#DatainMotionTour

Agenda
Develop a modern data strategy
Act on real time with data streaming
Build seamless streaming with AWS and Confluent
Innovate together to power customer success
Key take aways

Data volume and velocity
Multiple analytics needs
Variety of sources and data types
Difficult to manage systems
Complex to scale
Inflexible tools
Security, compliance
Slow performance Increasing and unpredictable cost
Challenges of data analytics at scale

Catalog
Governance
Analytics
Machine
learning Databases
Data
lakes
Modern data strategy
People,
apps and
devices
Data
sources

Modern data architecture on AWS
Key Pillars
Data at any
scale
Seamless data access and
movement
Purpose-built for best price performance
Built-in ML to solve business
challenges
Unified
governance

The benefits of data lakes
Catalog
Store all your data in open formats
Decouple storage from compute
Cost-effectively scale storage to exabytes
Process data in place
Choice of analytical and ML engines

Sharing across data movement with Data Mesh
Data producers Data mesh Data consumers
Unique modern data architecture
Suited to business function
Teams that want to share data
Unique modern data architecture
Suited to business function
Team that runs the marketplace Teams that want to use data

Source: Perishable insights, Mike Gualtieri, Forrester
Real time Seconds Minutes Hours Days Months
Value
of
data
to
decision-making
Preventive/predictive
Actionable Reactive Historical
Time-critical decisions Traditional “batch” business intelligence
Information half-life
in decision-making
Why real-time data streaming analytics ?
Data loses value quickly over time

Common real-time analytics use cases
Anomaly and fraud detection
Empowering IoT analytics
Nourishing marketing campaigns
Real-time personalization
Tailoring customer experience in real time
Supporting healthcare and emergency services

Challenges of data streaming
Difficult to set up Tricky to scale
Hard to achieve high availability Integration requires development
Error prone and complex to manage Expensive to maintain

Source
Devices and/or
applications
that produce
real-time
data at high
velocity
Stream ingestion
Data from tens of
thousands of data
sources can be written to
a single stream
In-stream storage
Data are stored in the
order they were received
for a set duration
of time and can be
replayed indefinitely
during that time
Stream processing
Records are read in
the order they are
produced, enabling real-
time analytics or streaming
ETL
Destination/
Storage
Database (NoSQL
most common)
Data lake
Data warehouse
Event driven
Applications
`
Visualize
Analytics
dashboard
Real-time streaming analytics pipeline
Ingest, Process & Analyze High Volumes of High-Velocity Data from Various Sources in Real
Time

Stream processing Visualize
Source
Mobile device
Metering
Click streams
IoT sensors
AWS SDKs
Kinesis Agent/KPL
Stream ingestion
`
Apache Kafka
Amazon Kinesis Data
Streams
Amazon Kinesis Data
Firehose
In-stream storage
Apache Kafka
Amazon Kinesis Data
Streams
Amazon Kinesis
Data Analytics
AWS Glue Streaming
Amazon EMR
Destination/Storage
Amazon DynamoDB
Amazon EMR
Amazon S3
Amazon Redshift Amazon OpenSearch
Amazon QuickSight
Apache Kafka
Streaming analytics on AWS

Rich front-end
customer
experiences
Real-time
Event Streams and Analysis
A Sale A shipment
A Trade
A Customer
Experience
Real-time backend
operations
Event streaming with Kafka to set Data in Motion:
Continuously processing evolving streams of data in real
time

Out-of-box integration with
popular services
Certified and validated by
AWS
AWS Native Services
Top-5 Global ISV for S3 Data Volume
3rd-Party ISV Services
• Amazon RDS Ready
• AWS Lambda Ready
• Amazon Redshift Ready
• AWS PrivateLink Ready
• AWS Outposts Ready
Validated Service Designations
Confluent integrations with AWS

S
Confluent Cloud on AWS – reference architecture
S3

Confluent + AWS: Accelerate Customer Business Outcomes
Topline Impacting New
Experiences
● Event-driven & real-time
● Unify data across org. w/ Kafka
data fabric (Schema Reg,..)
● AWS Analytics, Redshift, ML
connectors
Mitigate Risk
● Higher Service Quality &
Resilience with 99.99% SLA
● Deep Kafka expertise & innovation
● Elastic billing/pricing
Developer Agility
● Focus on innovation (not data
infrastructure)
● Leverage full Kafka OSS
ecosystem + AWS services
Faster Time to Market
● ~50-75% faster time to market*
● Streamline hybrid cloud
migration with no complex lift-n-
shift
● Maintain business continuity
Lower Kafka TCO
● ~25-50% lower TCO *
● GBps-scale & fast deployments
for global expansion
● Deploy Kafka at scale in 1 week
Maximize ROI
● ~200% ROI per Forrester study
● Save 10s of $Ms with legacy
offload to AWS with Confluent
Replicator
* For customers that don’t already have Kafka based system in-market
* TCO assessment to be analyzed for specific customer scenarios

Accelerate modernization from on-prem to AWS
Redshift Sink
Lambda Sink
AWS Direct
Connect
Replicator
LEGACY EDW
MAINFRAME
LEGACY DB
JDBC / CDC
connectors
Connect
Leverage 100+ Confluent pre-built connectors
Modernize
Value added apps, increase agility, reduce
TCO
On-prem AWS Cloud
Bridge
Hybrid cloud streaming
Amazon Athena
AWS Glue
SageMaker
Lake Formation
Amazon
DynamoDB
Amazon
Aurora
S3 Sink
Data Streams
Apps
ksqlDB

Confluent Wavelength solution with AWS

Increase developer agility & speed of innovation
Serverless integration
Connect for effortless integrations with Lambda & data stores
AWS serverless platform
Free up backend operation & Infras. management
Apps
Microservices
ksqlDB
Schema
Registry
COMPUTE
AWS
Lambda
Data stores
REST Proxy
& Clients
Source
Connectors
Lambda
Sink
DATA STORES
Amazon
DynamoDB
Amazon
Aurora
STORAGE
Amazon
S3
S3 Sink
ANALYTICS
Amazon
Athena
Amazon
Redshift

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Confluent
Data in Motion
23
Ahmed Zamzam
Sr Partner Solutions Engineer

...many more
Other
systems
Other
systems
Kafka Connect
Kafka Cluster
Kafka Connect
Apache Kafka is an event streaming platform
24

Core Kafka features
01
Publish & Subscribe
to streams of events
02
Store
your event streams
03
Process & Analyze
your events streams
26

ksqlDB
27

70%
of Fortune 500 companies
use Apache Kafka
(majority are Confluent customers)
The rise of data in motion
28

32,232
Stack overflow
questions
210
meetups
69,000
meetup attendees
10,286
Jira tickets for
Apache Kafka
46,626
Emails sent to
Apache Kafka mailing list
988
KIPs
Community
29

Hall of Innovation
CTO Innovation
Award Winner
2019
Enterprise Technology Innovation
AWARDS
Vision
● Original Kafka creators
founded Confluent
● Data in motion pioneers
Category leadership
● 80% of Kafka commits
● 5M+ hours of Kafka
technical experience
● Operate 35K+ clusters
Value
● Remove risk
● Deploy at scale
● Accelerate time to market
Product
● Extends Kafka to be
secure and
enterprise-ready
● Software or
cloud-native service
Confluent is the only company
focused on data in motion
30

Confluent: Everywhere
Confluent Platform
The Enterprise Distribution of
Apache Kafka
Confluent Cloud
Apache Kafka Re-engineered
for the Cloud
Self-Managed Software
Fully-Managed Service
VM
Deploy on any platform, on-prem or cloud
Available on the leading public clouds
31

Federated streaming, hybrid
and multi-cloud.
Data syndication and replication
across and between clouds and on-
premises, with self-service APIs, data
governance, and visual tooling.
Reliable & real-time data streams
between all customer sites, so you
can run always-on streaming
analytics on the data of the entire
enterprise, despite regional or cloud
provider outages.
Everywhere:
Cluster Linking Global Central Nervous System

K A F K A C L O U D
33

We Transformed Kafka for the Cloud, Ground Up!
Resilient
with automated
operations to ensure
high availability and
reliability
Performant
with networking
service decoupling
and replication
optimization
Elastic
to seamlessly
expand and shrink
based on customer
demands
KORA ENGINE
The Apache Kafka® Engine Built for the Cloud
Cost efficient
with multi-tenancy,
data tiering, cloud
optimizations and
hands-off operations

Event streaming as a cloud native service
Deploy in Minutes
Ensure
service levels
Integrate with cloud
services
Eliminate operational
burden
35

Scale faster with serverless development
Stream with Kafka
in minutes
Provision and scale
clusters on-demand
Pay only for
what you use
Start from $0 and pay based
on actual usage
36
Access latest
Kafka version
Always work in the most recent
stable version of Kafka

Go to market in 6 months vs. 2+ years
Traditional Kafka development
Development with Confluent Cloud
6-9 months to hire
Kafka resources
3 months
to ramp
Project
Kickoff
9-12 months build the production-grade
Kafka platform and develop application
Start
in 1
week
3-6 months to focus on app
development, not managing
Kafka
Project
Kickoff
Launch your application in months
and grow your business
Go to market in 2 years
Go to market in 6 months
37

38
Infinite Data

Infinite Apache Kafka
Scalable
Quickly scale up storage
resources without
overprovisioning infra
Efficient
Decouple storage from
compute resources to keep
storage costs low
Infinite
Retain as much data in Kafka
as you need to meet any use
case
39
Decouples the storage and compute layers to create highly scalable,
cost-effective storage without limits to enable infinite retention

Which use cases or
concerns does
Infinite address?
New customer experiences -
reimagine what you can do
with streaming through
infinite retention of events
Ease operational overhead -
stop worrying about running
out of disk space and simplify
topic configuration
40

41
Getting Data
into Kafka

Make Kafka
Widely Accessible
to Developers
Let developers build with their most
productive language and enable use
cases and architectures that require
multiple programming languages
Confluent Clients
Battle-tested and high performing

Apache Kafka Connect API:
Fault tolerant
Manage hundreds of
data sources and sinks
Preserves data schema
Integrated within
Confluent Control Center
JDBC
Mongo
MySQL
Elastic
HDFS
Kafka Connect API
Kafka Pipeline
Connector
Connector
Connector
Connector
Connector
Connector
Sources
Sinks
S3
43
Import and export data in and out of Kafka

140+
prebuilt
connectors
100+ Confluent supported 30+ partner supported, Confluent verified
AWS
Lambda
Instantly connect popular data sources & sinks
44

Confluent has deep AWS service integrations

46
Real-time
Transformations

Kafka Streams: Write standard Java apps and
microservices to process your data in real time
KStream<User, PageViewEvent> pageViews = builder.stream("pageviews-topic");
KTable<Windowed<User>, Long> viewsPerUserSession = pageViews
.groupByKey()
.count(SessionWindows.with(TimeUnit.MINUTES.toMillis(5)), "session-views");
docs.confluent.io/current/streams/
No separate processing cluster required
Develop on Mac, Linux, and Windows
Deploy to containers, VMs, bare metal, cloud
Powered by Kafka: elastic, scalable,
distributed, and reliable
Perfect for small, medium, and large use cases
Fully integrated with Kafka security
Exactly-once processing semantics
Part of Apache Kafka
47

ksqlDB at a glance
What is it?
ksqlDB is an event-streaming
database for working with
streams and tables of data
All the key features of a modern
streaming solution
Aggregations Joins
Windowing
Event-time
Dual query
support
Exactly-once
semantics
Out-of-order
handling
User-defined
functions
CREATE TABLE activePromotions AS
SELECT rideId,
qualifyPromotion(distanceToDst) AS
promotion
FROM locations
GROUP BY rideId
EMIT CHANGES
How does it work?
It separates compute from storage, and scales
elastically in a fault-tolerant manner
It remains highly available during disruption, even in
the face of failure to a quorum of its servers

3 modalities of stream processing with Confluent
Kafka clients Kafka Streams ksqlDB
ConsumerRecords<String, String> records = consumer.poll(100);
Map<String, Integer> counts = new DefaultMap<String, Integer>();
for (ConsumerRecord<String, Integer> record : records) {
String key = record.key();
int c = counts.get(key)
c += record.value()
counts.put(key, c)
}
for (Map.Entry<String, Integer> entry : counts.entrySet()) {
int stateCount;
int attempts;
while (attempts++ < MAX_RETRIES) {
try {
stateCount = stateStore.getValue(entry.getKey())
stateStore.setValue(entry.getKey(), entry.getValue() + stateCount)
break;
} catch (StateStoreException e) {
RetryUtils.backoff(attempts);
}
}
}
builder
.stream("input-stream",
Consumed.with(Serdes.String(), Serdes.String()))
.groupBy((key, value) -> value)
.count()
.toStream()
.to("counts", Produced.with(Serdes.String(), Serdes.Long()));
SELECT x, count(*) FROM stream GROUP BY x EMIT CHANGES;
Flexibility Simplicity
confluent.awsworkshop.io
49

Easily build event streaming applications
Use one, lightweight SQL
syntax to build a complete
real-time application
CREATE STREAM payments(user VARCHAR,
payment_amount INT)
WITH (kafka_topic = ’all_payments’,
key = ’user’,
value_format = ’avro’);
Create aggregations of
event data that can serve
queries to applications
Enrich Kafka data with a
robust stream processing
framework
USER Payment
Jay $10
Sue $15
Fred $5
... ...
USER Credit Score
Jay 660
Sue 710
Fred 595
USER Credit Score
Jay 660
Sue 710
Fred 595
USER Credit Score
Jay 660
Sue 710
Fred 595
50

How to get started
1. Subscribe to Confluent Cloud on the AWS Marketplace and start
with a free $400 (to be used within 60 days)
2. Try out the Confluent/AWS workshop found at
https://confluent.awsworkshop.io/
3. Email us at awsteam@confluent.io if you have any questions
51

Hands-on workshop,
and Q&A
52
Ahmed Zamzam
Sr Partner Solutions Engineer, Confluent

Conclusion/Next steps
• Thank you for attending.
• Please fill out the AWS event survey and let
us know what you thought.
• If you have any further questions or wish to
schedule a call with an AWS and Confluent
expert, please contact
awsteam@confluent.io
53

Confluent_AWS_ImmersionDay_Q42023.pdf

Recommandé

Recommandé

Contenu connexe

Similaire à Confluent_AWS_ImmersionDay_Q42023.pdf

Similaire à Confluent_AWS_ImmersionDay_Q42023.pdf (20)

Dernier

Dernier (20)

Confluent_AWS_ImmersionDay_Q42023.pdf