airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Laura Lorenz <llor...@industrydive.com>
Subject Re: external dag dependency: clear downstream task in another dag
Date Fri, 28 Oct 2016 13:11:37 GMT
I don't think so.

As I understand it the way the ExternalTaskSensor works is a simple,
pull-based dependency; its polls the metadata database until its associated
'external' TaskInstance has a state it expects (e.g. "SUCCESS"), so from
the ExternalTaskSensor's point of view a cleared 'external' TaskInstance,
not having the approved state yet, means it could just keep waiting forever
(until the timeout of course) and has no other connection. They don't have
any explicit `set_upstream()` or `set_downstream()` kind of relationship so
the upstream/downstream 'clear' or 'mark success' type of CLI traversal
won't find them.

On Wed, Oct 26, 2016 at 1:55 PM, Michael Gong <gonwg@hotmail.com> wrote:

> Hi,
>
> When I clear the downstream tasks of 1 task from 1 dag, can it also clear
> its downstream in another dag ?
>
>
> For example:
>
>
> dag1: task1
>
> dag2: task2 (which is an ExternalTaskSensor, and depends on task1)
>
>
> When I clear task1's downstream, is it possible that task2 is also cleared
> ?
>
>
>
> Any comments are welcomed.
>
>
> Thanks.
>
> Michael
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message