airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Chen <chingchien.c...@gmail.com>
Subject How to handle this case "Another instance is running, skipping" ?
Date Sun, 21 May 2017 01:53:03 GMT
Hi Airflow team,


I am using airflow with celery (2 nodes; i.e., two AWS instances)
My dag looks like below (the python dag name is task_ABC.py). Note in the
dag python file, I setup "max_active_runs=1"


           /---------> TaskB1 -----------> TaskC1---------\
TaskA -----------> TaskB2  ----------> TaskC2----------> TaskD
           \----------> TaskB3  -----------> TaskC3--------/

So, After TaskA; it runs TaskB1, TaskB2 and TaskB3 simultaneously.  TaskB1,
B2 and B3 are running same shell-script (TaskB.sh) with different input
arguments. It drops "Another instance is running, skipping" warning for
TaskB1 and TaskB3 (as the log below). It did not drop same warning in
TaskB2, I think it's because TaskB2 is running in different celery node (I
have two celery nodes).
If I manually make TaskB1 as successful, TaskB3 can proceed

The following is the log. Any idea to handle this ?
Thanks.

-Jason

========= Log of TaskB1 ============

[2017-05-20 23:09:47,270] {models.py:154} INFO - Filling up the DagBag
from /code/task_ABC.py
[2017-05-20 23:09:49,017] {models.py:154} INFO - Filling up the DagBag
from /code/task_ABC.py
[2017-05-20 23:09:49,165] {models.py:1196} INFO -
--------------------------------------------------------------------------------
Starting attempt 1 of 2
--------------------------------------------------------------------------------

[2017-05-20 23:09:49,182] {models.py:1219} INFO - Executing
<Task(PythonOperator): TaskB1> on 2017-05-20 03:40:00
[2017-05-20 23:09:49,214] {task_ABC.py:185} INFO -
/mycode/process/gfs0p25/TaskB.sh 2017052012 rain
[2017-05-21 00:09:56,054] {models.py:154} INFO - Filling up the DagBag
from /code/task_ABC.py
[2017-05-21 00:09:59,759] {models.py:154} INFO - Filling up the DagBag
from /code/task_ABC.py
[2017-05-21 00:10:00,008] {models.py:1146} WARNING - Another instance
is running, skipping.


========= Log of TaskB3 ============

[2017-05-20 23:09:44,660] {models.py:154} INFO - Filling up the DagBag
from /code/task_ABC.py
[2017-05-20 23:09:46,047] {models.py:154} INFO - Filling up the DagBag
from /code/task_ABC.py
[2017-05-20 23:09:46,205] {models.py:1196} INFO -
--------------------------------------------------------------------------------
Starting attempt 1 of 2
--------------------------------------------------------------------------------

[2017-05-20 23:09:46,224] {models.py:1219} INFO - Executing
<Task(PythonOperator): TaskB3> on 2017-05-20 03:40:00
[2017-05-20 23:09:46,257] {best_weather-BLEND-v1-1-0.py:245} INFO -
/mycode/process/gfs0p25/TaskB.sh 2017052012 snow
[2017-05-21 00:09:48,029] {models.py:154} INFO - Filling up the DagBag
from /code/task_ABC.py
[2017-05-21 00:09:49,080] {models.py:154} INFO - Filling up the DagBag
from /code/task_ABC.py
[2017-05-21 00:09:49,156] {models.py:1146} WARNING - Another instance
is running, skipping.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message