airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Yang <yrql...@gmail.com>
Subject Re: Blog post: Upgrading & Scaling Airflow at Robinhood
Date Fri, 09 Aug 2019 23:33:18 GMT
Nice post Abhishek! Glad our discussion was helpful for you guys.

To share more context with the community, Airbnb had task stuck in QUEUED
state problem before too. Our issues were more on the executor side.
Originally it was because message lost issue in early version celery, which
Alex Guziel applied an internal logic to resend tasks if they were not
picked up by the worker after 60s. Recently it was because of message
delivery delay issue on our Redis broker hosted on AWS Elasticache, which
we solved by moving to SQS( ya a sudden strange headache, attaching two
graphs to show the diff).

For the issue you guys have, which can be fixed by restarting the
scheduler, I took a quick look at the scheduler code and my wild guess of
the root cause is this piece of logic
<https://github.com/apache/airflow/blob/e5726c761d08bfddb6bb8acf3ecc381220eea140/airflow/jobs/scheduler_job.py#L962-L967>--from
my understanding the biggest thing of scheduler restart is that executor
state will be flushed, which is aligned with the behavior.

We also have similar scheduler health checks and restart logic, maybe my
peers can add more details later ;)

And thanks Max for sharing👍


Cheers,
Kevin Y


On Fri, Aug 9, 2019 at 2:17 PM Tao Feng <fengtao04@gmail.com> wrote:

> +1 Max, thanks for sharing!
>
> On Fri, Aug 9, 2019 at 2:05 PM Jarek Potiuk <Jarek.Potiuk@polidea.com>
> wrote:
>
> > +1
> >
> > On Fri, Aug 9, 2019 at 10:54 PM Maxime Beauchemin <
> > maximebeauchemin@gmail.com> wrote:
> >
> >> Thanks to Abhishek Ray @ Robinhood for this great post. I felt like I
> had
> >> to share it here
> >>
> >>
> https://robinhood.engineering/upgrading-scaling-airflow-at-robinhood-5b625dfaa2ee
> >>
> >> Max
> >>
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
> >
> >
>

Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message