airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Chen <chingchien.c...@gmail.com>
Subject Re: scheduler running on multiple nodes
Date Fri, 24 Feb 2017 19:04:04 GMT
A side question related to this topic:
I am running Airflow w/ celery executor in multiple nodes. Each node is
running celery, worker, scheduler and webserver.
These nodes are registered to a Redis for celery queue and these nodes are
sharing the same dags, logs folder (and MySQL)
It seems running fine.
Any concerns or suggestions ?
I am thinking celery executor is designed for distributed env.

Thanks.

-Jason


On Fri, Feb 24, 2017 at 10:58 AM, Jason Jho <jason.jho@blueapron.com.invalid
> wrote:

> Seems like this would inherently tied to the VM it's running on. Either
> way, would love to hear about any experiences as well!
> On Fri, Feb 24, 2017 at 1:52 PM Wilson Lian <wwlian@google.com.invalid>
> wrote:
>
> > Out of curiosity, has anyone heard any war stories re: reaching the
> limits
> > of a single scheduler in terms of the number of potentially-schedulable
> > DAGs?
> >
> > On Fri, Feb 24, 2017 at 10:25 AM, Dan Davydov <
> > dan.davydov@airbnb.com.invalid> wrote:
> >
> > > We just had two running by accident for some period of time.
> > >
> > > On Feb 24, 2017 5:52 AM, "Jason Jho" <jason.jho@blueapron.com.invalid>
> > > wrote:
> > >
> > > > Hi Dan / Sid,
> > > >
> > > > Would you be able to elaborate on the multiple scheduler setup?
> Curious
> > > how
> > > > that would have been deployed. Was the purpose to have some kind of
> > > > failover or to distribute execution of jobs?
> > > >
> > > > Thanks!
> > > > On Fri, Feb 24, 2017 at 3:49 AM Dan Davydov <dan.davydov@airbnb.com.
> > > > invalid>
> > > > wrote:
> > > >
> > > > > Fwiw Airbnb was running multiple schedulers for a short while on
> > 1.7.1
> > > > and
> > > > > we didn't seem to have issues.
> > > > >
> > > > > On Feb 24, 2017 12:25 AM, "Bolke de Bruin" <bdbruin@gmail.com>
> > wrote:
> > > > >
> > > > > > While I agree with the assessment of Sid that a lot has changed
> and
> > > we
> > > > do
> > > > > > not officially test on multiple schedulers, many changes were
in
> > the
> > > > area
> > > > > > of proper locking which benefit multiple schedulers. In addition
> > the
> > > > > tasks
> > > > > > themselves have built in checks that they don’t run twice
at the
> > same
> > > > > time.
> > > > > >
> > > > > > Yet YMMV.
> > > > > >
> > > > > > Bolke
> > > > > >
> > > > > > > On 24 Feb 2017, at 03:13, siddharth anand <sanand@apache.org>
> > > wrote:
> > > > > > >
> > > > > > > I did  run 2 or more schedulers with Local Executors up
until
> mid
> > > > last
> > > > > > > year. There have been enough changes to the code and feature
> > > > additions
> > > > > > that
> > > > > > > I don't think this is a recommended practice at this point.
> Also,
> > > > there
> > > > > > is
> > > > > > > not a lot of synchronization in the scheduler to ensure
this
> will
> > > > work.
> > > > > > >
> > > > > > > -s
> > > > > > >
> > > > > > > On Thu, Feb 9, 2017 at 6:47 AM, matus valo <
> matusvalo@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > >> Hi all,
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> I am considering deployment of airflow as pipeline
framework.
> I
> > > have
> > > > > > found
> > > > > > >> out multiple articles explaining deployment of airflow
in
> > > > distributed
> > > > > > >> environment (e.g. [1]). Unfortunately, I was not able
to find
> > out
> > > > any
> > > > > > use
> > > > > > >> case where scheduler is deployed distributed on multiple
> nodes.
> > Is
> > > > it
> > > > > > >> possible to have scheduler distributed on multiple
nodes to
> > > prevent
> > > > > > single
> > > > > > >> point of failure? I haven’t found any mention about
it in
> > > > > > documentation. I
> > > > > > >> have found out in [2] that it is not possible but on
the other
> > > hand
> > > > in
> > > > > > [3]
> > > > > > >> is reference that this can be solved in new version
of
> airflow.
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >>
> > > > > > >>
> > > > > > >> Matus
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> [1] http://site.clairvoyantsoft.com/setting-apache-airflow-
> > > cluster/
> > > > > > >>
> > > > > > >> [2]
> > > > > https://groups.google.com/forum/#!topic/airbnb_airflow/-1wKa3OcwME
> > > > > > >>
> > > > > > >> [3] https://issues.apache.org/jira/browse/AIRFLOW-678
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message