airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Froetscher <rfroetsc...@lumoslabs.com>
Subject Re: external task dependency across 2 different servers
Date Thu, 17 Nov 2016 18:52:54 GMT
Are you saying you have multiple installations of Airflow, or simply that
your airflow setup is distributed, with celery workers on their own nodes?

If the latter, ExternalTaskSensor should still do the trick for you, as I
believe it just checks airflow's operational database for the status of
this external task.

On Thu, Nov 17, 2016 at 10:40 AM, Michael Gong <gonwg@hotmail.com> wrote:

> Hi,
>
>
> Is it possible that 1 task can depend on another task running on another
> DAG on another sever?
>
>
> For example, I have task1 in dag1, and  task2 in dag2.
>
>
> If both dag1 and dag2 running on the same airflow instance on the same
> server, airflow provides the ExternalTaskSensor operator.
>
>
> But if dag2 runs on another airflow instance on another sever, can airflow
> allow dag1's task1 depend on dag2's task2?
>
>
>
> Any suggestions are welcomed.
>
>
> thanks
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message