airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Davydov <dan.davy...@airbnb.com.INVALID>
Subject Re: scheduler running on multiple nodes
Date Fri, 24 Feb 2017 18:25:11 GMT
We just had two running by accident for some period of time.

On Feb 24, 2017 5:52 AM, "Jason Jho" <jason.jho@blueapron.com.invalid>
wrote:

> Hi Dan / Sid,
>
> Would you be able to elaborate on the multiple scheduler setup? Curious how
> that would have been deployed. Was the purpose to have some kind of
> failover or to distribute execution of jobs?
>
> Thanks!
> On Fri, Feb 24, 2017 at 3:49 AM Dan Davydov <dan.davydov@airbnb.com.
> invalid>
> wrote:
>
> > Fwiw Airbnb was running multiple schedulers for a short while on 1.7.1
> and
> > we didn't seem to have issues.
> >
> > On Feb 24, 2017 12:25 AM, "Bolke de Bruin" <bdbruin@gmail.com> wrote:
> >
> > > While I agree with the assessment of Sid that a lot has changed and we
> do
> > > not officially test on multiple schedulers, many changes were in the
> area
> > > of proper locking which benefit multiple schedulers. In addition the
> > tasks
> > > themselves have built in checks that they don’t run twice at the same
> > time.
> > >
> > > Yet YMMV.
> > >
> > > Bolke
> > >
> > > > On 24 Feb 2017, at 03:13, siddharth anand <sanand@apache.org> wrote:
> > > >
> > > > I did  run 2 or more schedulers with Local Executors up until mid
> last
> > > > year. There have been enough changes to the code and feature
> additions
> > > that
> > > > I don't think this is a recommended practice at this point. Also,
> there
> > > is
> > > > not a lot of synchronization in the scheduler to ensure this will
> work.
> > > >
> > > > -s
> > > >
> > > > On Thu, Feb 9, 2017 at 6:47 AM, matus valo <matusvalo@gmail.com>
> > wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >>
> > > >>
> > > >> I am considering deployment of airflow as pipeline framework. I have
> > > found
> > > >> out multiple articles explaining deployment of airflow in
> distributed
> > > >> environment (e.g. [1]). Unfortunately, I was not able to find out
> any
> > > use
> > > >> case where scheduler is deployed distributed on multiple nodes. Is
> it
> > > >> possible to have scheduler distributed on multiple nodes to prevent
> > > single
> > > >> point of failure? I haven’t found any mention about it in
> > > documentation. I
> > > >> have found out in [2] that it is not possible but on the other hand
> in
> > > [3]
> > > >> is reference that this can be solved in new version of airflow.
> > > >>
> > > >>
> > > >>
> > > >> Thanks,
> > > >>
> > > >>
> > > >> Matus
> > > >>
> > > >>
> > > >>
> > > >> [1] http://site.clairvoyantsoft.com/setting-apache-airflow-cluster/
> > > >>
> > > >> [2]
> > https://groups.google.com/forum/#!topic/airbnb_airflow/-1wKa3OcwME
> > > >>
> > > >> [3] https://issues.apache.org/jira/browse/AIRFLOW-678
> > > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message