airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: external task dependency across 2 different servers
Date Fri, 18 Nov 2016 09:47:28 GMT
Looking ahead forward to a not too distant future: I’m currently finishing up the foundations
for a Rest API. It would be trivial to add a call that returns the state of a task in this
API. Then having a federated or even fully decoupled HA becomes a possibility. Obviously it
is experimental at the moment cause it will need to settle a bit.

https://github.com/apache/incubator-airflow/pull/1783

Bolke

> Op 17 nov. 2016, om 22:47 heeft Rob Froetscher <rfroetscher@lumoslabs.com> het
volgende geschreven:
> 
> Hey Michael,
> 
> If you have separate installations with separate operational databases,
> then the ExternalTaskSensor will not do what you need. However, I'm not
> sure I understand why you would need separate Airflow installations that
> rely upon each other. Using Celery, you should be able to scale a single
> installation of Airflow as much as you need.
> 
> Regarding setting up celery worker nodes. There is some information on that
> in the Airflow Documentation
> <https://airflow.incubator.apache.org/configuration.html#scaling-out-with-celery>
> .
> 
> On Thu, Nov 17, 2016 at 11:48 AM, Michael Gong <gonwg@hotmail.com> wrote:
> 
>> Rob,
>> 
>> thanks for reply.
>> 
>> 
>> If I have multiple installations of airflow on different servers, will
>> this dependency be done ?
>> 
>> 
>> For the second setup, ie. 1 installation of airflow and it is distributed,
>> can you advise how to set it up ? I am not familiar with that.
>> 
>> 
>> Thanks.
>> 
>> 
>> 
>> ________________________________
>> From: Rob Froetscher <rfroetscher@lumoslabs.com>
>> Sent: Thursday, November 17, 2016 6:52 PM
>> To: dev@airflow.incubator.apache.org
>> Subject: Re: external task dependency across 2 different servers
>> 
>> Are you saying you have multiple installations of Airflow, or simply that
>> your airflow setup is distributed, with celery workers on their own nodes?
>> 
>> If the latter, ExternalTaskSensor should still do the trick for you, as I
>> believe it just checks airflow's operational database for the status of
>> this external task.
>> 
>> On Thu, Nov 17, 2016 at 10:40 AM, Michael Gong <gonwg@hotmail.com> wrote:
>> 
>>> Hi,
>>> 
>>> 
>>> Is it possible that 1 task can depend on another task running on another
>>> DAG on another sever?
>>> 
>>> 
>>> For example, I have task1 in dag1, and  task2 in dag2.
>>> 
>>> 
>>> If both dag1 and dag2 running on the same airflow instance on the same
>>> server, airflow provides the ExternalTaskSensor operator.
>>> 
>>> 
>>> But if dag2 runs on another airflow instance on another sever, can
>> airflow
>>> allow dag1's task1 depend on dag2's task2?
>>> 
>>> 
>>> 
>>> Any suggestions are welcomed.
>>> 
>>> 
>>> thanks
>>> 
>>> 
>>> 
>> 


Mime
View raw message