airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joy Gao <j...@wepay.com>
Subject Re: DAG with ExternalTaskSensor won't run after deletion
Date Tue, 12 Sep 2017 19:10:39 GMT
Hi Ali,

I was able to replicate your scenario above.

The executor maintains a cache of running task instances. When a task
instance finishes, it is removed from the `running` cache of the executor.
When you hard-delete the DAG Run and the task instances via the UI, the
corresponding task instances are still in the 'running' cache. So when the
scheduler is iterating through the dags, it is told by the executor that
this particular task instance is already running, and does not need to be
rescheduled.  If you bounce the scheduler, the task should be picked up
again as the cache will be regenerated.

I believe this is a bug in the scheduler. So feel free to report it as a
JIRA issue on this. Although in general,  I try to avoid hard-deleting
items from the DB when they are in `running` state. Hope this helps!

Joy

On Mon, Sep 11, 2017 at 2:38 PM, Ali Uz <aliuz1@gmail.com> wrote:

> Hello,
>
> I have a DAG that has an ExternalTaskSensor that waits for another DAG to
> complete before running. Here's my scenario: I run this DAG and while the
> external task sensor is running, I pause it, then delete all its tasks and
> the DAG Run from the Admin panel. Then I turn it on again, and the DAG
> displays as "running" in the UI, but none of its tasks get queued.
>
> Also in the Jobs UI there seems to be a Job stuck in "running" which seems
> to be related to the DAG I just deleted.
>
> How can I ensure that my DAG will always run no matter what?
>
> Thanks,
>
> - Ali
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message