airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Chen <chingchien.c...@gmail.com>
Subject Questions on Airflow with CeleryExecutor
Date Wed, 02 Nov 2016 20:24:41 GMT
Hi Airflow team,

We are using Airflow with LocalExecutor and it works great.
We are moving toward to use CeleryExecutor and have couple of questions.
I searched the posts and cannot find some answers.

We have 3 airflow worker nodes and uses Redis as broker.

(1) How airflow worker determines the next task should be dispatched to ?
Is it based on the average load of the nodes ? How long the task runs
previously ?
(2) In my settings of 3 nodes, I have webserver, scheduler and worker (w/
Celery) running on each node. Is it a recommended setting (i.e., each
service runs on each node) ?
(3) When a task runs and fails, I have a on_failure_callback. Will the
on_failure_callback runs on the same node as the task ? Or, the executor is
based on task level (seems so) ?
(4) There is setting for the queue (default_queue = default) in
airflow.cfg.
Say, I want to use different queues to control which worker should use
which queue. I think just change default_queue = other_queue. Right ?
Should I do any additional Celery related setting ?
(5) If all workers are done and schedulers are still running, will the
tasks be queued ?
(6) If we turned of the tasks, while there are tasks in the queue
previously,  will it kill the un-needed tasks in the queue ?

Thanks.
Jason

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message