airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From George Leslie-Waksman <geo...@cloverhealth.com.INVALID>
Subject Re: How to handle this case "Another instance is running, skipping" ?
Date Thu, 25 May 2017 20:54:41 GMT
That is the expected behavior. What happened is two worker processes each
grabbed a request of TaskB1, but one of them noticed, and left it to the
other one.

There are a handful of reasons it might be showing up in your logs.

Which version of Airflow are you running? Is your scheduler set to restart
periodically? Are you running more than one scheduler?

On Sat, May 20, 2017 at 6:53 PM Jason Chen <chingchien.chen@gmail.com>
wrote:

> Hi Airflow team,
>
>
> I am using airflow with celery (2 nodes; i.e., two AWS instances)
> My dag looks like below (the python dag name is task_ABC.py). Note in the
> dag python file, I setup "max_active_runs=1"
>
>
>            /---------> TaskB1 -----------> TaskC1---------\
> TaskA -----------> TaskB2  ----------> TaskC2----------> TaskD
>            \----------> TaskB3  -----------> TaskC3--------/
>
> So, After TaskA; it runs TaskB1, TaskB2 and TaskB3 simultaneously.  TaskB1,
> B2 and B3 are running same shell-script (TaskB.sh) with different input
> arguments. It drops "Another instance is running, skipping" warning for
> TaskB1 and TaskB3 (as the log below). It did not drop same warning in
> TaskB2, I think it's because TaskB2 is running in different celery node (I
> have two celery nodes).
> If I manually make TaskB1 as successful, TaskB3 can proceed
>
> The following is the log. Any idea to handle this ?
> Thanks.
>
> -Jason
>
> ========= Log of TaskB1 ============
>
> [2017-05-20 23:09:47,270] {models.py:154} INFO - Filling up the DagBag
> from /code/task_ABC.py
> [2017-05-20 23:09:49,017] {models.py:154} INFO - Filling up the DagBag
> from /code/task_ABC.py
> [2017-05-20 23:09:49,165] {models.py:1196} INFO -
>
> --------------------------------------------------------------------------------
> Starting attempt 1 of 2
>
> --------------------------------------------------------------------------------
>
> [2017-05-20 23:09:49,182] {models.py:1219} INFO - Executing
> <Task(PythonOperator): TaskB1> on 2017-05-20 03:40:00
> [2017-05-20 23:09:49,214] {task_ABC.py:185} INFO -
> /mycode/process/gfs0p25/TaskB.sh 2017052012 <(201)%20705-2012> rain
> [2017-05-21 00:09:56,054] {models.py:154} INFO - Filling up the DagBag
> from /code/task_ABC.py
> [2017-05-21 00:09:59,759] {models.py:154} INFO - Filling up the DagBag
> from /code/task_ABC.py
> [2017-05-21 00:10:00,008] {models.py:1146} WARNING - Another instance
> is running, skipping.
>
>
> ========= Log of TaskB3 ============
>
> [2017-05-20 23:09:44,660] {models.py:154} INFO - Filling up the DagBag
> from /code/task_ABC.py
> [2017-05-20 23:09:46,047] {models.py:154} INFO - Filling up the DagBag
> from /code/task_ABC.py
> [2017-05-20 23:09:46,205] {models.py:1196} INFO -
>
> --------------------------------------------------------------------------------
> Starting attempt 1 of 2
>
> --------------------------------------------------------------------------------
>
> [2017-05-20 23:09:46,224] {models.py:1219} INFO - Executing
> <Task(PythonOperator): TaskB3> on 2017-05-20 03:40:00
> [2017-05-20 23:09:46,257] {best_weather-BLEND-v1-1-0.py:245} INFO -
> /mycode/process/gfs0p25/TaskB.sh 2017052012 <(201)%20705-2012> snow
> [2017-05-21 00:09:48,029] {models.py:154} INFO - Filling up the DagBag
> from /code/task_ABC.py
> [2017-05-21 00:09:49,080] {models.py:154} INFO - Filling up the DagBag
> from /code/task_ABC.py
> [2017-05-21 00:09:49,156] {models.py:1146} WARNING - Another instance
> is running, skipping.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message