airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxime Beauchemin <maximebeauche...@gmail.com>
Subject Re: scheduler running on multiple nodes
Date Fri, 24 Feb 2017 21:01:59 GMT
One drawback is the lack of proper cross DAG prioritization. Note that in
>= 1.8:

* the scheduler is a multi process queue, in terms of fire power it should
not be the bottleneck for anyone at this point. Effectively you could have
64+ scheduling threads on a single large machine. This should make the db
the new bottleneck, but serve the needs of any large company doing
something reasonable (from my guts I'd say firing thousands of tasks per
minute should work smoothly)
* each process in the queue evaluates a single DAG and reports runnable
tasks to the main scheduler process which accumulates, prioritizes and puts
messages in the queue on a specified interval. If you have DAGs that are
slow to process for some reason, they'll miss the train get on board of the
next one.
* where double-triggers are normally an edge case, they become the norm
when running multiple schedulers
* in theory, double triggers should be handled properly, and fire once and
only once. But we've seen occurrences of double-firing in 1.8.0, which
we're fixing now, but minimizing double triggers helps lowering the risk of
double-firing.
* note that tasks may be indempotent from a sequential perspective but
double-firing in parallel may have bad side-effects
* the existing protections and light guarantees around double-firing, along
with luck around "parallely idempotent" tasks is what mitigated
double-triggers and made running multiple schedulers work historically

For Airflow to have a "features distributed scheduling" stamp on it, we'd
have to somehow prevent most double-triggering when running multiple
schedulers. This could be achieved by the individual schedulers somehow
taking locks, perhaps at the DAG level (impacts cross DAG prioritization in
pools negatively), or by taking turns on running scheduling cycles somehow
(would enable resiliency only, not distributing the workload).

Ideally, I think we'd want to accumulate runnable tasks in a message queue,
and schedulers would only take turns on the process that read from that
queue, prioritizes and send messages to run tasks on workers.

Max

On Fri, Feb 24, 2017 at 11:51 AM, Bolke de Bruin <bdbruin@gmail.com> wrote:

> Can I assume your db is also not a SPOF?
>
> It seems a waste of processor time to me to have it every node.
>
> B.
>
> Sent from my iPhone
>
> > On 24 Feb 2017, at 20:39, Jason Chen <chingchien.chen@gmail.com> wrote:
> >
> > Thanks, Arthur.
> >
> > I think it avoids "single point of failure" (and maybe balance the
> > scheduling load between nodes ?)
> > Given a celery cluster, if the only node running scheduler is down, the
> > whole cluster will fail to schedule jobs.
> > Any downside why not having multiple schedulers ?
> >
> > -Jason
> >
> >
> > On Fri, Feb 24, 2017 at 11:35 AM, Arthur Wiedmer <
> arthur.wiedmer@gmail.com>
> > wrote:
> >
> >> Jason,
> >>
> >> Why do you need a scheduler running on each node?
> >>
> >> We have a single scheduler powering work on many nodes each running a
> >> celery worker via "airflow worker". We have one large metadata MySQL
> >> instance.
> >>
> >> Best,
> >> Arthur
> >>
> >> On Fri, Feb 24, 2017 at 11:04 AM, Jason Chen <chingchien.chen@gmail.com
> >
> >> wrote:
> >>
> >>> A side question related to this topic:
> >>> I am running Airflow w/ celery executor in multiple nodes. Each node is
> >>> running celery, worker, scheduler and webserver.
> >>> These nodes are registered to a Redis for celery queue and these nodes
> >> are
> >>> sharing the same dags, logs folder (and MySQL)
> >>> It seems running fine.
> >>> Any concerns or suggestions ?
> >>> I am thinking celery executor is designed for distributed env.
> >>>
> >>> Thanks.
> >>>
> >>> -Jason
> >>>
> >>>
> >>> On Fri, Feb 24, 2017 at 10:58 AM, Jason Jho <jason.jho@blueapron.com.
> >>> invalid
> >>>> wrote:
> >>>
> >>>> Seems like this would inherently tied to the VM it's running on.
> Either
> >>>> way, would love to hear about any experiences as well!
> >>>> On Fri, Feb 24, 2017 at 1:52 PM Wilson Lian <wwlian@google.com.invalid
> >>>
> >>>> wrote:
> >>>>
> >>>>> Out of curiosity, has anyone heard any war stories re: reaching
the
> >>>> limits
> >>>>> of a single scheduler in terms of the number of
> >> potentially-schedulable
> >>>>> DAGs?
> >>>>>
> >>>>> On Fri, Feb 24, 2017 at 10:25 AM, Dan Davydov <
> >>>>> dan.davydov@airbnb.com.invalid> wrote:
> >>>>>
> >>>>>> We just had two running by accident for some period of time.
> >>>>>>
> >>>>>> On Feb 24, 2017 5:52 AM, "Jason Jho" <jason.jho@blueapron.com.
> >>> invalid>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hi Dan / Sid,
> >>>>>>>
> >>>>>>> Would you be able to elaborate on the multiple scheduler
setup?
> >>>> Curious
> >>>>>> how
> >>>>>>> that would have been deployed. Was the purpose to have some
kind
> >> of
> >>>>>>> failover or to distribute execution of jobs?
> >>>>>>>
> >>>>>>> Thanks!
> >>>>>>> On Fri, Feb 24, 2017 at 3:49 AM Dan Davydov <
> >>> dan.davydov@airbnb.com.
> >>>>>>> invalid>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Fwiw Airbnb was running multiple schedulers for a short
while
> >> on
> >>>>> 1.7.1
> >>>>>>> and
> >>>>>>>> we didn't seem to have issues.
> >>>>>>>>
> >>>>>>>> On Feb 24, 2017 12:25 AM, "Bolke de Bruin" <bdbruin@gmail.com>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>>> While I agree with the assessment of Sid that a
lot has
> >> changed
> >>>> and
> >>>>>> we
> >>>>>>> do
> >>>>>>>>> not officially test on multiple schedulers, many
changes were
> >>> in
> >>>>> the
> >>>>>>> area
> >>>>>>>>> of proper locking which benefit multiple schedulers.
In
> >>> addition
> >>>>> the
> >>>>>>>> tasks
> >>>>>>>>> themselves have built in checks that they don’t
run twice at
> >>> the
> >>>>> same
> >>>>>>>> time.
> >>>>>>>>>
> >>>>>>>>> Yet YMMV.
> >>>>>>>>>
> >>>>>>>>> Bolke
> >>>>>>>>>
> >>>>>>>>>> On 24 Feb 2017, at 03:13, siddharth anand <
> >> sanand@apache.org
> >>>>
> >>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> I did  run 2 or more schedulers with Local Executors
up
> >> until
> >>>> mid
> >>>>>>> last
> >>>>>>>>>> year. There have been enough changes to the
code and
> >> feature
> >>>>>>> additions
> >>>>>>>>> that
> >>>>>>>>>> I don't think this is a recommended practice
at this point.
> >>>> Also,
> >>>>>>> there
> >>>>>>>>> is
> >>>>>>>>>> not a lot of synchronization in the scheduler
to ensure
> >> this
> >>>> will
> >>>>>>> work.
> >>>>>>>>>>
> >>>>>>>>>> -s
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Feb 9, 2017 at 6:47 AM, matus valo <
> >>>> matusvalo@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi all,
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> I am considering deployment of airflow as
pipeline
> >>> framework.
> >>>> I
> >>>>>> have
> >>>>>>>>> found
> >>>>>>>>>>> out multiple articles explaining deployment
of airflow in
> >>>>>>> distributed
> >>>>>>>>>>> environment (e.g. [1]). Unfortunately, I
was not able to
> >>> find
> >>>>> out
> >>>>>>> any
> >>>>>>>>> use
> >>>>>>>>>>> case where scheduler is deployed distributed
on multiple
> >>>> nodes.
> >>>>> Is
> >>>>>>> it
> >>>>>>>>>>> possible to have scheduler distributed on
multiple nodes
> >> to
> >>>>>> prevent
> >>>>>>>>> single
> >>>>>>>>>>> point of failure? I haven’t found any
mention about it in
> >>>>>>>>> documentation. I
> >>>>>>>>>>> have found out in [2] that it is not possible
but on the
> >>> other
> >>>>>> hand
> >>>>>>> in
> >>>>>>>>> [3]
> >>>>>>>>>>> is reference that this can be solved in
new version of
> >>>> airflow.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Matus
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> [1] http://site.clairvoyantsoft.
> >> com/setting-apache-airflow-
> >>>>>> cluster/
> >>>>>>>>>>>
> >>>>>>>>>>> [2]
> >>>>>>>> https://groups.google.com/forum/#!topic/airbnb_airflow/-
> >>> 1wKa3OcwME
> >>>>>>>>>>>
> >>>>>>>>>>> [3] https://issues.apache.org/jira/browse/AIRFLOW-678
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message