This document provides an overview of Amundsen, an open source data discovery and metadata platform developed by Lyft. It begins with an introduction to the challenges of data discovery and outlines Amundsen's architecture, which uses a graph database and search engine to provide metadata about data resources. The document discusses how Amundsen impacts users at Lyft by reducing time spent searching for data and discusses the project's community and future roadmap.
5. Data is used to make informed decisions
5
Analysts Data Scientists General
Managers
Engineers ExperimentersProduct
Managers
Data-driven decision making process:
1. Search & find data
2. Understand the data
3. Perform an analysis/visualisation
4. Share insights and/or make a decision
Make data the heart of every decision
6. • Goal: What new data-driven policies can we enact to reduce driver
insurance fraud?
• Idea: Let’s take a deeper look into insurance claims from drivers who
have given less than 𝑥 rides.
• Next Step: I’ll first get all drivers who have given less than 𝑥 rides...but
where do I look?
Hi! I’m a new Analyst in the Fraud Department !
6
7. • Ask a friend/manager/coworker
• Ask in a wider Slack channel
• Search in the Github repos
Step 1: Search & find data
7
We end up finding tables: driver_rides
& rides_driver_total
8. • What is the difference: driver_rides vs. rides_driver_total
• What do the different fields mean?
‒ Is driver_rides.completed different from
rides_driver_total.lifetime_completed?
‒ What period of time does the data in each table cover?
• Dig deeper: explore using SQL queries
Step 2: Understand the data
8
SELECT * FROM schema.driver_rides
WHERE ds=’2019-05-15’
LIMIT 100;
SELECT * FROM schema.rides_driver_total
WHERE ds=’2019-05-15’
LIMIT 100;
9. - No way to know &
understand trusted data
- Created channels & oncalls
for data questions
Lots of queries like:
SELECT
*
FROM
default.my_table
WHERE ds=’2018-01-01’
LIMIT 100;
Lack of productivity had many side effects
9
- Does data exist?
- Prior work?
- Source of truth?
- Who owns it?
- Who uses it?
Lots of unknowns Increased database load Interrupt heavy data culture
10. Lots of wasted tech & biz users time
10
Analyst/DS workflow and time spent on each step
12. Holy grail of solving for productivity
12
metadata
noun /ˈmedəˌdādə,ˈmedəˌdadə/
:a set of data that describes and gives information about other data.
1. What kind of
information?
2. About what
data?
13. 1. What kind of information? (aka ABC of metadata)
13
Application Context
Metadata needed by humans or applications to operate
● Where is the data?
● What are the semantics of the data?
Behavior
How is data created and used over time?
● Who’s using the data?
● Who created the data?
Change
Change in data over time
● How is the data evolving over time?
● Evolution of code that generates the data Terminology borrowed from Ground paper
14. Short answer: Any data within your organization
Long answer:
2. About what data?
14
Data stores Dashboard /
Reports
Schema registry
Events /
Schemas
Streams People
Employees
15. 3 complementary ways to do Data Discovery
15
Search based
I am looking for a table with data on “cancel rates”
- Where is the table?
- What does it contain?
- Has the analysis I want to perform already been done?
Lineage based
If this event is down, what datasets are going to be impacted?
- Upstream/downstream lineage
- Incidents, SLA misses, Data quality
Network based
I want to check what tables my manager uses
- Ownership information
- Bookmarking
- Usage through query logs
16. Data discovery for ALL users
16
Power User
- Has been at Lyft for a long time
- Knows the data environment well:
where to find data, what it means,
how to use it
Pain points:
- Needs to spend a fair amount of
their time sharing their knowledge
with the new user
- Could become “New user” if they
switch teams
New User
- Recently joined Lyft or switched to
a new team
- Needs to ramp up on a lot of
things, wants to start having impact
soon
Pain points:
- Doesn’t know where to start.
Spends their time asking questions
and cmd+F on github
- Makes mistakes by mis-using
some datasets
Other requirements
- Leverage as much data automatically as possible
- Preferably, open source and healthy community
- API availability
- Easy to set up
17. Solution space
• Vendors - Alation, Collibra
• Existing open source projects (e.g. Apache Atlas)
• LinkedIn’s data portal - Wherehows & DataHub (blog, code)
• Twitter’s data discovery (blog)
• Netflix’s metacat (code, blog)
• Airbnb’s data portal (blog, video)
• Big Query SQL Web UI & catalog (blog)
• Goods: Organizing Google’s Datasets (paper)
• Data Warehousing and Analytics Infrastructure at Facebook (paper) 17
18. Compared various existing solutions/open source projects
Criteria / Products Alation Where
Hows
Airbnb
Data
Portal
Cloudera
Navigator
Apache
Atlas
Search based
Lineage based
Network based
Hive/Presto support
Redshift support
Open source (pref.)
19. Amundsen
19
Product named after Roald Amundsen
● First expedition to reach the South Pole
● First to explore both North & South Poles
33. 33
Postgres Hive Redshift ... Presto
Github
Source
File
Databuilder Crawler
Neo4j
Elastic
Search
Metadata Service Search Service
Frontend ServiceML
Feature
Service
Security
Service
Other Microservices
Metadata Sources
34. 34
Metadata Service
• A thin proxy layer to interact with graph database
‒ Currently Neo4j is the default option for graph backend engine
‒ Work with the community to support Apache Atlas
• Support Rest API for other services pushing / pulling metadata directly
41. Metadata - Challenges
• No Standardization: No single data model that fits for all data
resources
‒ A data resource could be a table, an Airflow DAG or a dashboard
• Different Extraction: Each data set metadata is stored and fetched
differently
‒ Hive Table: Stored in Hive metastore
‒ RDBMS(postgres etc): Fetched through DBAPI interface
‒ Github source code: Fetched through git hook
‒ Mode dashboard: Fetched through Mode API
‒ …
41
44. 44
Postgres Hive Redshift ... Presto
Github
Source
File
Databuilder Crawler
Neo4j
Elastic
Search
Metadata Service Search Service
Frontend ServiceML
Feature
Service
Security
Service
Other Microservices
Metadata Sources
45. Search Service
• A thin proxy layer to interact with the search backend
‒ Currently it supports Elasticsearch as the search backend.
• Support different search patterns
‒ Normal Search: match records based on relevancy
‒ Category Search: match records first based on data type, then
relevancy
‒ Wildcard Search
45
46. How to make the search result more relevant?
46
• Collect metrics
‒ Instrumentation for search behavior
‒ Measure click-through-rate (CTR) over top 5 results
• Experiment with different weights, e.g boost the exact table ranking
• Advanced search:
‒ Support wildcard search (e.g. event_*)
‒ Support category search (e.g. column: is_line_ride)
‒ Future: Filtering, Autosuggest
52. 52
“This is God’s
work” - George
X, ex-head of
Analytics, Lyft
“I was on call and
I’m confident 50%
of the questions
could have been
answered by a
simple search in
Amundsen” -
Bomee P, DS,
Lyft
A6n @ Lyft
56. Recent Contributions from the community
• BigQuery integration (Coolblue)
• PostgreSQL and Redshift integration (Everfi)
• Security improvements and Apache Atlas integration (ING)
• Snowflake integration (LMC)
• Toolbar on landing page (In progress, Workday)
• Integrating with Delta analytics platform (In progress, Databricks)
• Talks by ING & Coolblue at conferences in Barcelona, Vilnius & Moscow
56
60. Roadmap (subject to change, not ordered)
• Index Dashboards (Product spec)
• Link business terms and process to technical metadata
• Standardize Information Governance metadata
• Include tags in search
• ACL integration, allow only specific roles to edit descriptions
• Show search context for what matched
• “Request for descriptions” aka notifications
• Data Lineage
60
61. Phil Mizrahi | @philippemizrahi | in/philippe-mizrahi
Project Code @ github.com/lyft/amundsen
Blog Post @ go.lyft.com/datadiscoveryblog
Icons under Creative Commons License from https://thenounproject.com/
61