CIS Cyber Data Lake Analytics

P U B L I C S E C T O R
S U M M I T
WASH INGTON DC

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
Cyber Data Lake: How CIS Analyzes
Billions of Network Traffic Records
per Day
Brian Calkin
Chief Technology Officer
Center for Internet Security
3 0 2 6 3 9
Oliver Atoa
Senior Consultant
AWS/WWPS Proserve
Bob Strahan
Principal Consultant
AWS/WWPS Proserve

S U M M I T
Agenda
1. Center for Internet Security (CIS) Netflow Challenge (Brian)
2. Our Solution (Bob and Oliver)
3. Results (Brian)
4. Q&A

S U M M I T
Session Goals
1. Educate you with useful architecture concepts
2. Empower you to explore similar approaches for your own business
Familiarity with Data Lakes on AWS presumed
– this not a Data Lake pitch!

S U M M I T

S U M M I T
Center for Internet Security
Non-profit using the power of a global
community to develop best practices for
securing IT systems and data
Mission: Identify, develop, validate,
promote, and sustain best practice solutions
for cyber defense. Build and lead
communities to enable an environment of
trust in cyberspace.

S U M M I T
CIS Benchmarks and CIS Controls
CIS Benchmarks and CIS Controls are the global standard
and recognized best practices for securing IT systems and data
against the most pervasive attacks
CIS’ proven guidelines are continuously refined and verified by a
volunteer, global community of experienced IT professionals

S U M M I T
Multi-State & Elections Infrastructure
Information Analysis Center (MS-ISAC & EI-ISAC)
The MS-ISAC has been designated by DHS as the key
resource for cyber threat prevention, protection, response,
and recovery for the nation’s state, local, tribal, and
territorial governments
Through the EI-ISAC, election agencies gain access to an
elections-focused cyber defense suite, including sector-
specific threat intelligence products, incident response and
remediation, threat and vulnerability monitoring,
cybersecurity awareness and training products, and tools for
implementing security best practices.
~6,000 Member
Organizations

S U M M I T
CIS Network Monitoring - Albert
• Network intrusion detection
sensor
• Fully monitored and managed
• State, local, tribal and territorial
government focused
• Open source software on
commodity hardware
• Alert data analyzed 24x7
• ~350 Sensors deployed
nationwide

S U M M I T
• Generated on sensors
• Passive DNS data also
collected
• Data is valuable for performing
ad-hoc queries
Albert - NetFlow
• Source IP
• Destination IP
• Source Port
• Destination Port
• TCP Flags
• Number of bytes of traffic
sent/received
• Timestamp

S U M M I T
• Based on Suricata
• Alerts generated based on
known signatures
• ~27,000 Signatures per
Sensor
• ~10,000 Albert events
analyzed per month
• ~5,000 Albert events escalated
to SLTT entities per month
Albert – Network Intrusion Detection

S U M M I T
The Challenge
- 48 Million Records Per Minute
- Rate of incoming traffic is not consistent and continually
increasing
- Several petabytes (and growing) of NetFlow data
- Local SAN Storage Full
- Ad-hoc queries take way too long to run
- Hours, Days, Weeks…

S U M M I T
The Desired Solution
- Store Six Months of NetFlow Data
- High Performance Ad-Hoc SQL Queries
- Goal: Get from Days and Weeks to Seconds and Minutes
- Cost Effective
- Highly Secure
- Extensible Features for Future Enhancements and
Growth
*Looked at Both On-Premise & Cloud Solutions

S U M M I T
Our mission
Ingest binary netflow and DPI records from hundreds of sensors
each generating millions of records every couple of minutes.

S U M M I T
Our mission
.
Transform and enrich these records, and save them to cost
efficient storage.

S U M M I T
Our mission
Provide analysts with fast seamless SQL query
access to all the records; newest (few minutes latency) to oldest
(months back).

S U M M I T
Our mission
Make it secure, cost effective, well instrumented,
reliable, scalable to handle future growth, and extensible as a
foundation for advanced automated analytics and machine
learning.

S U M M I T
Data Lake

S U M M I T
Ingest binary netflow and DPI records from hundreds of sensors each
generating millions of records every couple of minutes.
• Receiver service on AWS
• Amazon Elastic Compute Cloud (Amazon EC2)
Autoscaling, Multi-AZ, NLB with EIP
• SCP file transfer with client auth (keypairs)
• IP whitelisting
• Receivers convert each incoming sensor file
to CSV files, and uploads each file to
Amazon Simple Storage Service (Amazon S3)
(not yet enriched or query ready)
• Receiver cluster scales up and down as
incoming record volumes fluctuate.
Ingestion

S U M M I T
Enrichment and Amazon S3 data lake storage
Transform and enrich flow records and save them to cost efficient storage.
We use a AWS Lambda function
• Triggered immediately as each new CSV
file arrives in Amazon S3 from Receiver
• For each record:
• Detect corrupted records and fix or reject
them
• Enrich good records with additional useful
fields (e.g. IP ASN, directionality, etc.)
• Save records to Amazon S3 (stage0), with
prefixes that define Hive partitions:
• p_sensor=<sensorname>/p_year=YYYY/
p_date=YYYY-MM-DD/p_hour=YYYY-MM-
DD_hh/<file>.csv.gz

S U M M I T
Near real-time SQL access
Provide analysts with fast seamless SQL query access to all the records;
newest (few minutes latency) to oldest (months back).
Enrich AWS Lambda function also adds new
partitions to predefined AWS Glue catalog
tables
Partitions optimize query cost and speed; filters
on sensorname and/or flow timestamp use
‘partition pruning’
Records are accessible to analysts via Amazon
Athena SQL
Latency (sensor to SQL) < 5min

S U M M I T
Optimize for efficient long term queries
Provide analysts with fast seamless SQL query access to all the records;
newest (few minutes latency) to oldest (months back).
Small files (micro-batches) minimize the
latency for NRT queries (stage0).
But we need to optimize for large time
span queries (stage1)
Stage 1 table has same columns and
partitions as stage0, but:
• Columnar file format (parquet)
• Bucketed by srcIP (faster queries)
• Minimize files per hourly partition
Deep dive coming up!

S U M M I T
Seamless SQL access
Provide analysts with fast seamless SQL query access to all the records
Recap
• Stage 0 optimized for NRT queries (Enrich Lambda)
– lots of small files per partition
• Stage 1 optimized for historical queries (ETL)
– fewer optimized files per partition
Views combine (UNION ALL) stage0 & stage1
tables to give the best of both worlds
• Low latency for recent data from stage0 (<6hrs)
• Efficiency (cost + speed) from stage1 (>6hrs)
• Views updated every hour (scheduled Lambda)

S U M M I T
Well Architected!
Make it secure, cost effective, well instrumented, reliable,
scalable to handle future growth, and extendible as a foundation
for advanced automated analytics and machine learning.
Data Lake on Amazon S3 +
AWS Glue catalog: foundation for
future enhancements and
innovations (using the ‘right tool for
the job’ – Amazon SageMaker,
Amazon QuickSight, Amazon
RedShift, etc.)
Automated
Deployment:
AWS CodePipeline,
AWS CodeBuild, and
AWS CloudFormation
Storage Retention & Costs:
Amazon S3 Intelligent Tiering and
Lifecycle Management

S U M M I T
Stage 1 ETL – Deep Dive
• Columnar file format - Parquet
• Bucketed by source IP (fixed
number of larger files)
• CSV
• Variable number of
small files in NRT
Make queries faster and cheaper

S U M M I T
VPC
AWS Cloud
Auto Scaling group
EC2 instance EC2 instance
Stage0 puts partition values in
ETL Amazon Simple Queue
Service
(Amazon SQS) queue
AutoScaling Group driven by
Amazon SQS queue depth
Each Amazon EC2 instance runs
a Dask script that processes one
partition at a time - avoid
network shuffling
Maintain Stage0 partition
structure
ETL SQS Queue
Partition Message
Read all CSV files in partition
Data Lake Bucket
Data catalog
stage0
stage1
Partition Message
Write bucketed Parquet
Get table info & update partition

S U M M I T
Why not Amazon Kinesis or Firehose?
Initially iterated using established streaming patterns with
Kinesis and Firehose. As we optimized, we needed greater
control for this particular case:
• Avoid data shuffling. The file micro-batches being ingested
are inherently partitioned in the way we needed
• Firehose partitions files in Amazon S3 by ingestion time. To
optimize NRT queries we need to partition by fields in the
payload: e.g. sensorname and flow timestamp
• File format optimizations. E.g. Parquet row group size and
Hive Bucketing
Amazon Simple
Queue Service
Amazon EC2
Amazon
Kinesis
Amazon Kinesis
Data Streams
Amazon Kinesis
Data Firehose

S U M M I T
Why Dask and not Amazon EMR, Spark, or AWS Glue?
Initially iterated using AWS Glue and Amazon EMR. Awesome tools!
Landed on Dask for greater low-level control based on a combination
of optimizations and features for this particular case:
• Skewed partitions – larger sensors can cause stragglers
• Reduce network data shuffling
• Amazon S3 multi-part uploads. Avoid staging and visible when
successful
• Automatic retry of failed partitions
• Dynamic partition overwrite. Backfill or redo of partitions
• Partition management – make them available when needed
• Hive Bucketing
Amazon
EMR
AWS Glue
Amazon EC2

S U M M I T
Bucketing
• Hive Partitions for small set of values
• Hive Bucketing for large unique values
(e.g., IP address billions)
• Bucketing provides significant
performance improvement for queries
using bucketed fields
• Hash field and apply mod function based
on number of buckets
• Fixed number of evenly distributed files
• We implemented it in Dask hash(bucket) %
number of buckets

S U M M I T
• Partitioning targets a
specific folder
• Bucketing targets a
specific file
• Columnar format can
skip data within the file

S U M M I T
The results are in…
The answer
is…
You may be asking yourself, did it all work out??
Yes!

S U M M I T
Some Examples
Count of all records generated by a single
sensor
• The “old way” – 15 minutes
• The AWS way – 2 minutes
7.5x
faster
Count of all records generated by all sensors
• Old way – 36 hours
• AWS – 3 minutes
720x faster

S U M M I T
Athena Query 1 – Flows Byte Aggregation 1HR

S U M M I T
A couple more…
Query for all traffic, destined to a specific IP
address and port, over a one week time
period
150x faster
Query for all traffic, destined to a set of IP
addresses over port 80, over a one week
time period
360x faster

S U M M I T
Athena Query 2 – All Traffic to a Single IP & Port – 1 Week

S U M M I T
Next Steps
• Leverage Amazon Redshift Spectrum
• Building UI for broader usability
• Collect additional datatypes for improved alerting and correlation
• Leverage Artificial Intelligence/Machine Learning to identify malicious
activity

S U M M I T
Thank you!
S U M M I T
Bob Strahan
strahanr@amazon.com
Oliver Atoa
oatoa@amazon.com
Brian Calkin
Brian.Calkin@cisecurity.org

S U M M I T
S U M M I T

CIS Cyber Data Lake Analytics

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à CIS Cyber Data Lake Analytics

Similaire à CIS Cyber Data Lake Analytics (20)

Plus de Amazon Web Services

Plus de Amazon Web Services (20)

CIS Cyber Data Lake Analytics