airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Chen <chingchien.c...@gmail.com>
Subject Re: scheduler running on multiple nodes
Date Fri, 24 Feb 2017 19:39:58 GMT
Thanks, Arthur.

 I think it avoids "single point of failure" (and maybe balance the
scheduling load between nodes ?)
 Given a celery cluster, if the only node running scheduler is down, the
whole cluster will fail to schedule jobs.
 Any downside why not having multiple schedulers ?

-Jason


On Fri, Feb 24, 2017 at 11:35 AM, Arthur Wiedmer <arthur.wiedmer@gmail.com>
wrote:

> Jason,
>
> Why do you need a scheduler running on each node?
>
> We have a single scheduler powering work on many nodes each running a
> celery worker via "airflow worker". We have one large metadata MySQL
> instance.
>
> Best,
> Arthur
>
> On Fri, Feb 24, 2017 at 11:04 AM, Jason Chen <chingchien.chen@gmail.com>
> wrote:
>
> > A side question related to this topic:
> > I am running Airflow w/ celery executor in multiple nodes. Each node is
> > running celery, worker, scheduler and webserver.
> > These nodes are registered to a Redis for celery queue and these nodes
> are
> > sharing the same dags, logs folder (and MySQL)
> > It seems running fine.
> > Any concerns or suggestions ?
> > I am thinking celery executor is designed for distributed env.
> >
> > Thanks.
> >
> > -Jason
> >
> >
> > On Fri, Feb 24, 2017 at 10:58 AM, Jason Jho <jason.jho@blueapron.com.
> > invalid
> > > wrote:
> >
> > > Seems like this would inherently tied to the VM it's running on. Either
> > > way, would love to hear about any experiences as well!
> > > On Fri, Feb 24, 2017 at 1:52 PM Wilson Lian <wwlian@google.com.invalid
> >
> > > wrote:
> > >
> > > > Out of curiosity, has anyone heard any war stories re: reaching the
> > > limits
> > > > of a single scheduler in terms of the number of
> potentially-schedulable
> > > > DAGs?
> > > >
> > > > On Fri, Feb 24, 2017 at 10:25 AM, Dan Davydov <
> > > > dan.davydov@airbnb.com.invalid> wrote:
> > > >
> > > > > We just had two running by accident for some period of time.
> > > > >
> > > > > On Feb 24, 2017 5:52 AM, "Jason Jho" <jason.jho@blueapron.com.
> > invalid>
> > > > > wrote:
> > > > >
> > > > > > Hi Dan / Sid,
> > > > > >
> > > > > > Would you be able to elaborate on the multiple scheduler setup?
> > > Curious
> > > > > how
> > > > > > that would have been deployed. Was the purpose to have some
kind
> of
> > > > > > failover or to distribute execution of jobs?
> > > > > >
> > > > > > Thanks!
> > > > > > On Fri, Feb 24, 2017 at 3:49 AM Dan Davydov <
> > dan.davydov@airbnb.com.
> > > > > > invalid>
> > > > > > wrote:
> > > > > >
> > > > > > > Fwiw Airbnb was running multiple schedulers for a short
while
> on
> > > > 1.7.1
> > > > > > and
> > > > > > > we didn't seem to have issues.
> > > > > > >
> > > > > > > On Feb 24, 2017 12:25 AM, "Bolke de Bruin" <bdbruin@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > While I agree with the assessment of Sid that a lot
has
> changed
> > > and
> > > > > we
> > > > > > do
> > > > > > > > not officially test on multiple schedulers, many changes
were
> > in
> > > > the
> > > > > > area
> > > > > > > > of proper locking which benefit multiple schedulers.
In
> > addition
> > > > the
> > > > > > > tasks
> > > > > > > > themselves have built in checks that they don’t
run twice at
> > the
> > > > same
> > > > > > > time.
> > > > > > > >
> > > > > > > > Yet YMMV.
> > > > > > > >
> > > > > > > > Bolke
> > > > > > > >
> > > > > > > > > On 24 Feb 2017, at 03:13, siddharth anand <
> sanand@apache.org
> > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > I did  run 2 or more schedulers with Local Executors
up
> until
> > > mid
> > > > > > last
> > > > > > > > > year. There have been enough changes to the code
and
> feature
> > > > > > additions
> > > > > > > > that
> > > > > > > > > I don't think this is a recommended practice
at this point.
> > > Also,
> > > > > > there
> > > > > > > > is
> > > > > > > > > not a lot of synchronization in the scheduler
to ensure
> this
> > > will
> > > > > > work.
> > > > > > > > >
> > > > > > > > > -s
> > > > > > > > >
> > > > > > > > > On Thu, Feb 9, 2017 at 6:47 AM, matus valo <
> > > matusvalo@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Hi all,
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> I am considering deployment of airflow as
pipeline
> > framework.
> > > I
> > > > > have
> > > > > > > > found
> > > > > > > > >> out multiple articles explaining deployment
of airflow in
> > > > > > distributed
> > > > > > > > >> environment (e.g. [1]). Unfortunately, I
was not able to
> > find
> > > > out
> > > > > > any
> > > > > > > > use
> > > > > > > > >> case where scheduler is deployed distributed
on multiple
> > > nodes.
> > > > Is
> > > > > > it
> > > > > > > > >> possible to have scheduler distributed on
multiple nodes
> to
> > > > > prevent
> > > > > > > > single
> > > > > > > > >> point of failure? I haven’t found any mention
about it in
> > > > > > > > documentation. I
> > > > > > > > >> have found out in [2] that it is not possible
but on the
> > other
> > > > > hand
> > > > > > in
> > > > > > > > [3]
> > > > > > > > >> is reference that this can be solved in new
version of
> > > airflow.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> Thanks,
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> Matus
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> [1] http://site.clairvoyantsoft.
> com/setting-apache-airflow-
> > > > > cluster/
> > > > > > > > >>
> > > > > > > > >> [2]
> > > > > > > https://groups.google.com/forum/#!topic/airbnb_airflow/-
> > 1wKa3OcwME
> > > > > > > > >>
> > > > > > > > >> [3] https://issues.apache.org/jira/browse/AIRFLOW-678
> > > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message