airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wilson Lian <wwl...@google.com.INVALID>
Subject Re: scheduler running on multiple nodes
Date Fri, 24 Feb 2017 18:52:22 GMT
Out of curiosity, has anyone heard any war stories re: reaching the limits
of a single scheduler in terms of the number of potentially-schedulable
DAGs?

On Fri, Feb 24, 2017 at 10:25 AM, Dan Davydov <
dan.davydov@airbnb.com.invalid> wrote:

> We just had two running by accident for some period of time.
>
> On Feb 24, 2017 5:52 AM, "Jason Jho" <jason.jho@blueapron.com.invalid>
> wrote:
>
> > Hi Dan / Sid,
> >
> > Would you be able to elaborate on the multiple scheduler setup? Curious
> how
> > that would have been deployed. Was the purpose to have some kind of
> > failover or to distribute execution of jobs?
> >
> > Thanks!
> > On Fri, Feb 24, 2017 at 3:49 AM Dan Davydov <dan.davydov@airbnb.com.
> > invalid>
> > wrote:
> >
> > > Fwiw Airbnb was running multiple schedulers for a short while on 1.7.1
> > and
> > > we didn't seem to have issues.
> > >
> > > On Feb 24, 2017 12:25 AM, "Bolke de Bruin" <bdbruin@gmail.com> wrote:
> > >
> > > > While I agree with the assessment of Sid that a lot has changed and
> we
> > do
> > > > not officially test on multiple schedulers, many changes were in the
> > area
> > > > of proper locking which benefit multiple schedulers. In addition the
> > > tasks
> > > > themselves have built in checks that they don’t run twice at the same
> > > time.
> > > >
> > > > Yet YMMV.
> > > >
> > > > Bolke
> > > >
> > > > > On 24 Feb 2017, at 03:13, siddharth anand <sanand@apache.org>
> wrote:
> > > > >
> > > > > I did  run 2 or more schedulers with Local Executors up until mid
> > last
> > > > > year. There have been enough changes to the code and feature
> > additions
> > > > that
> > > > > I don't think this is a recommended practice at this point. Also,
> > there
> > > > is
> > > > > not a lot of synchronization in the scheduler to ensure this will
> > work.
> > > > >
> > > > > -s
> > > > >
> > > > > On Thu, Feb 9, 2017 at 6:47 AM, matus valo <matusvalo@gmail.com>
> > > wrote:
> > > > >
> > > > >> Hi all,
> > > > >>
> > > > >>
> > > > >>
> > > > >> I am considering deployment of airflow as pipeline framework.
I
> have
> > > > found
> > > > >> out multiple articles explaining deployment of airflow in
> > distributed
> > > > >> environment (e.g. [1]). Unfortunately, I was not able to find
out
> > any
> > > > use
> > > > >> case where scheduler is deployed distributed on multiple nodes.
Is
> > it
> > > > >> possible to have scheduler distributed on multiple nodes to
> prevent
> > > > single
> > > > >> point of failure? I haven’t found any mention about it in
> > > > documentation. I
> > > > >> have found out in [2] that it is not possible but on the other
> hand
> > in
> > > > [3]
> > > > >> is reference that this can be solved in new version of airflow.
> > > > >>
> > > > >>
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >>
> > > > >> Matus
> > > > >>
> > > > >>
> > > > >>
> > > > >> [1] http://site.clairvoyantsoft.com/setting-apache-airflow-
> cluster/
> > > > >>
> > > > >> [2]
> > > https://groups.google.com/forum/#!topic/airbnb_airflow/-1wKa3OcwME
> > > > >>
> > > > >> [3] https://issues.apache.org/jira/browse/AIRFLOW-678
> > > > >>
> > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message