airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From siddharth anand <san...@apache.org>
Subject Re: scheduler running on multiple nodes
Date Fri, 24 Feb 2017 02:13:14 GMT
I did  run 2 or more schedulers with Local Executors up until mid last
year. There have been enough changes to the code and feature additions that
I don't think this is a recommended practice at this point. Also, there is
not a lot of synchronization in the scheduler to ensure this will work.

-s

On Thu, Feb 9, 2017 at 6:47 AM, matus valo <matusvalo@gmail.com> wrote:

> Hi all,
>
>
>
> I am considering deployment of airflow as pipeline framework. I have found
> out multiple articles explaining deployment of airflow in distributed
> environment (e.g. [1]). Unfortunately, I was not able to find out any use
> case where scheduler is deployed distributed on multiple nodes. Is it
> possible to have scheduler distributed on multiple nodes to prevent single
> point of failure? I haven’t found any mention about it in documentation. I
> have found out in [2] that it is not possible but on the other hand in [3]
> is reference that this can be solved in new version of airflow.
>
>
>
> Thanks,
>
>
> Matus
>
>
>
> [1] http://site.clairvoyantsoft.com/setting-apache-airflow-cluster/
>
> [2] https://groups.google.com/forum/#!topic/airbnb_airflow/-1wKa3OcwME
>
> [3] https://issues.apache.org/jira/browse/AIRFLOW-678
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message