This presentation covers how to setup an Airflow instance as a cluster which spans multiple machines instead of the traditional 1 machine distribution. In addition, it covers an added step you can take to ensure High Availability in that cluster.
3. 3Page:
Airflow Daemons
• Web Server
• Daemon that runs the Airflow Webserver
• 1 to many gunicorn processes to accept and process requests in
parallel.
• Allows you to track jobs progress, run jobs and more
• Scheduler
• Periodically runs (every X seconds) to determine if a DAG or Task
needs to be ran based off the DAG schedule
• Pushes messages to the Queuing Service to be executed
• Worker
• Daemon runs if you’re using the CeleryExecutors (as opposed to
SequentialExecutor and LocalExecutor)
• 1 to many dedicated celeryd processes which execute functions
• Pulls messages from a Queuing services to determine what
functions to execute
6. 6Page:
Why setup a Cluster Deployment?
• Distributes heavy processes onto many machines for better
use of resources
• More Highly Available Airflow environment
• If you have many Workflows with many Tasks your executors
would not be able to get to all the messages in the queue.
Adding more executors would fix this issue.
7. 7Page:
Scaling Workers
• Horizontally
• Add more machines to the cluster
• No need to register the machines with the master. You
just need to start up the Airflow Worker task on the new
Machine.
• Vertically
• Increase the number of executors (celeryd processes)
per node and restart the workers
9. 9Page:
Limitations
• There can only be one scheduler running at a time
• If you have multiple Scheduler processes running, there's
a possibility that multiple instances of a single task that
will be scheduled to run.
• If the Scheduler Daemon or Machine with the process goes
down then no jobs will get scheduled
10. 10Page:
Airflow Scheduler Failover Controller
• Dedicated Daemon that runs with Airflow on the Master
Nodes
• Ensures that there is always one and only one Scheduler
running on the Master nodes at a time
• Developed Internally and Open Sourced
• https://github.com/teamclairvoyant/airflow-scheduler-
failover-controller
• High Level Steps
• Polls (every x seconds) to check if the scheduler is
running
• If scheduler isn’t running, restart the scheduler
• If it still doesn’t start up, then try starting it up on the
other master nodes
13. 13Page:
Failover Controller Process (Start Up)
Master Node 1
Failover
Controller
(standby)
Master Node 2
Failover
Controller
(standby)
On startup, the processes start out in STANDBY
14. 14Page:
Failover Controller Process (Start Up)
Master Node 1
Failover
Controller
(active)
Master Node 2
Failover
Controller
(standby)
The first one to enter data into the Metastore is elected as the active
controller.
15. 15Page:
Failover Controller Process (Start Up)
Scheduler
Master Node 1
Failover
Controller
(active)
Master Node 2
Failover
Controller
(standby)
The Failover controller checks to see if the Scheduler is running, but it
isn’t.
16. 16Page:
Failover Controller Process (Start Up)
Scheduler
Master Node 1
Failover
Controller
(active)
Master Node 2
Failover
Controller
(standby)
Failover Controller starts up the Scheduler
21. 21Page:
Failover Controller Process (Process Failure 2)
Scheduler
Master Node 1
Failover
Controller
(active)
Master Node 2
Failover
Controller
(standby)
Scheduler process has died
22. 22Page:
Failover Controller Process (Process Failure 2)
Scheduler
Master Node 1
Failover
Controller
(active)
Master Node 2
Failover
Controller
(standby)
Failover Controller tries to restart the Scheduler, but its still not running
23. 23Page:
Failover Controller Process (Process Failure 2)
Scheduler
Master Node 1
Failover
Controller
(active)
Master Node 2
Failover
Controller
(standby)
Failover Controller tries to restart the Scheduler on a different node
24. 24Page:
Failover Controller Process (Process Failure 2)
Scheduler
Master Node 1
Failover
Controller
(active)
Master Node 2
Failover
Controller
(standby)
Failover Controller succeeds to restart the scheduler and the cluster is
back to normal
26. 26Page:
Failover Controller Process (Node Failure)
Scheduler
Master Node 1
Failover
Controller
(active)
Master Node 2
Failover
Controller
(standby)
Everything is running as expected
27. 27Page:
Failover Controller Process (Node Failure)
Scheduler
Master Node 1
Failover
Controller
(dead)
Master Node 2
Failover
Controller
(standby)
Master Node 1 dies and all the processes running on it are gone
28. 28Page:
Failover Controller Process (Node Failure)
Scheduler
Master Node 1
Failover
Controller
(dead)
Master Node 2
Failover
Controller
(active)
Failover Controller on Master 2 becomes active because the one running
on Master Node 1 has stopped sending a heart beat
29. 29Page:
Failover Controller Process (Node Failure)
Scheduler
Master Node 1
Failover
Controller
(dead)
Master Node 2
Failover
Controller
(active)
The newly active Failover Controller tries to check-in with and restart the
Scheduler on the daemon the Metadata says its running on and fails.
30. 30Page:
Failover Controller Process (Node Failure)
Scheduler
Master Node 1
Failover
Controller
(dead)
Master Node 2
Failover
Controller
(active)
The Failover Controller then starts it on another node and it succeeds
Scheduler
31. 31Page:
Failover Controller Process (Node Failure)
Master Node 1
Failover
Controller
(standby)
Master Node 2
Failover
Controller
(active)
When Master Node 1 is brought back, the old Failover Controller goes
into STANDBY state
Scheduler