Learn how to use PySpark for processing massive amounts of data. Combined with the GitHub repo - https://github.com/rdempsey/pyspark-for-data-processing - this presentation will help you gain familiarity with processing data using Python and Spark.
If you're thinking about machine learning and not sure if it can help improve your business, but want to find out, set up a free 20-minute consultation with us: https://calendly.com/robertwdempsey/free-consultation
4. The Three Jobs
At Atlantic Dominion Solutions we perform three functions for
our customers:
Consulting: we assess and advise in the areas of technology,
team and process to determine how machine learning can have
the biggest impact on your business.
Implementation: after a strategy session to determine the work
you need we get to work using our proven methodology and
begin delivering smarter applications.
Training: continuous improvement requires continuous learning.
We provide both on-premises and online training.
5. Co-authoring the book Building
Machine Learning Pipelines.
Written for software developers and
data scientists, Building Machine
Learning Pipelines teaches the skills
required to create and use the
infrastructure needed to run
modern intelligent systems.
machinelearningpipelines.com
Writing the Book
6. Robert Dempsey, CEO
Software Engineer
Books and online courses
Lotus Guides, District Data Labs
Atlantic Dominion Solutions, LLC
Professional
Author
Instructor
Owner
9. 1. When acquiring knowledge start by going wide instead of
deep.
2. Always focus on what's important to people rather than
just the technology.
3. Be able to clearly communicate what you know with
others.
Core Principles
18. Got Clean Air?
• Clean air is important.
• Toxic pollutants are known or suspected of causing cancer,
reproductive effects, birth defects, and adverse
environmental effects.
19. Questions to Answer
1. Which state has the highest level of pollutants?
2. Which county has the highest level of pollutants?
3. What are the top 5 pollutants by unit of measure?
4. What are the trends of pollutants by state over time?
24. The Core of Spark
• Computational engine that schedules, distributes and
monitors computational tasks running on a cluster
25. Higher Level Tools
• Spark SQL: SQL and structured data
• MLlib: machine learning
• GraphX: graph processing
• Spark Streaming: process streaming data
26. Storage
• Local file system
• Amazon S3
• Cassandra
• Hive
• HBase
• File formats
• Text files
• Sequence files
• Avro
• Parquet
• Hadoop Input Format
27. Hadoop?
• Not necessary, but…
• If you have multiple nodes you need a resource manager
like YARN or Mesos
• You'll need access to distributed storage like HDFS,
Amazon S3 or Cassandra
29. What Is PySpark?
• An API that exposes the Spark programming model to
Python
• Build on top of Spark's Java API
• Data is processed with Python and cached/shuffled in the
JVM
• Driver programs
30. Driver Programs
• Launch parallel operations on a cluster
• Contain application functions
• Define distributed datasets
• Access Spark through a SparkContext
• Uses Py4J to launch a JVM and create a
JavaSparkContext
31. When to Use It
• When you need to…
• Process boat loads of data (TB)
• Perform operations that require all the data to be in
memory (machine learning)
• Efficiently process streaming data
• Create an overly complicated use case to present at a
meetup
33. Docker
• Software container platform
• Containers are application only (no OS)
• Deployed anywhere with same CPU architecture (x86-64,
ARM)
• Available for *nix, Mac, Windows
47. Dive In
• Data explorer notebook
• Q1 - Most polluted state
• Q2 - Most polluted county
• Q3 - Top pollutants by unit of measure
• Q4 - Pollutants over time