airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: scheduler running on multiple nodes
Date Fri, 24 Feb 2017 08:25:03 GMT
While I agree with the assessment of Sid that a lot has changed and we do not officially test
on multiple schedulers, many changes were in the area of proper locking which benefit multiple
schedulers. In addition the tasks themselves have built in checks that they don’t run twice
at the same time.

Yet YMMV.

Bolke

> On 24 Feb 2017, at 03:13, siddharth anand <sanand@apache.org> wrote:
> 
> I did  run 2 or more schedulers with Local Executors up until mid last
> year. There have been enough changes to the code and feature additions that
> I don't think this is a recommended practice at this point. Also, there is
> not a lot of synchronization in the scheduler to ensure this will work.
> 
> -s
> 
> On Thu, Feb 9, 2017 at 6:47 AM, matus valo <matusvalo@gmail.com> wrote:
> 
>> Hi all,
>> 
>> 
>> 
>> I am considering deployment of airflow as pipeline framework. I have found
>> out multiple articles explaining deployment of airflow in distributed
>> environment (e.g. [1]). Unfortunately, I was not able to find out any use
>> case where scheduler is deployed distributed on multiple nodes. Is it
>> possible to have scheduler distributed on multiple nodes to prevent single
>> point of failure? I haven’t found any mention about it in documentation. I
>> have found out in [2] that it is not possible but on the other hand in [3]
>> is reference that this can be solved in new version of airflow.
>> 
>> 
>> 
>> Thanks,
>> 
>> 
>> Matus
>> 
>> 
>> 
>> [1] http://site.clairvoyantsoft.com/setting-apache-airflow-cluster/
>> 
>> [2] https://groups.google.com/forum/#!topic/airbnb_airflow/-1wKa3OcwME
>> 
>> [3] https://issues.apache.org/jira/browse/AIRFLOW-678
>> 


Mime
View raw message