airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russell Pierce <russell.s.pie...@gmail.com>
Subject Re: Tasks Queued but never run
Date Wed, 07 Jun 2017 15:34:37 GMT
Depending on how fast you can clear down your queue, -n can be harmful and
really stack up your celery queue. Keep an eye on your queue depth of you
see a ton of messages about the task already having been run.

On Mon, Jun 5, 2017, 9:18 AM Josef Samanek <josef.samanek@kiwi.com> wrote:

> Hey. Thanks for the answer. I previously also tried to run scheduler -n
> 10, but it was back when I was still using LocalExecutor. And it did not
> help. I have not yet tried to do it with CeleryExecutor, so I might.
>
> Still, I would prefer to find an actual solution for the underlying
> problem, not just a workaround (eventhough a working workaround is also
> appreciated).
>
> Best regards,
> Joe
>
> On 2017-06-02 00:10 (+0200), Alex Guziel <alex.guziel@airbnb.com.INVALID>
> wrote:
> > We've noticed this with celery, relating to this
> > https://github.com/celery/celery/issues/3765
> >
> > We also use `-n 5` option on the scheduler so it restarts every 5 runs,
> > which will reset all queued tasks.
> >
> > Best,
> > Alex
> >
> > On Thu, Jun 1, 2017 at 2:18 PM, Josef Samanek <josef.samanek@gmail.com>
> > wrote:
> >
> > > Hi!
> > >
> > > We have a problem with our airflow. Sometimes, several tasks get queued
> > > but they never get run and remain in Queud state forever. Other tasks
> from
> > > the same schedule interval run. And next schedule interval runs
> normally
> > > too. But these several tasks remain queued.
> > >
> > > We are using Airflow 1.8.1. Currently with CeleryExecutor and redis,
> but
> > > we had the same problem with LocalExecutor as well (actually switching
> to
> > > Celery helped quite a bit, the problem now happens way less often, but
> > > still it happens). We have 18 DAGs total, 13 active. Some have just 1-2
> > > tasks, but some are more complex, like 8 tasks or so and with
> upstreams.
> > > There are also ExternalTaskSensor tasks used.
> > >
> > > I tried playing around with DAG configurations (limiting concurrency,
> > > max_active_runs, ...), tried switching off some DAGs completely (not
> all
> > > but most) etc., so far nothing helped. Right now, I am not really sure,
> > > what else to try to identify a solve the issue.
> > >
> > > I am getting a bit desperate, so I would really appreciate any help
> with
> > > this. Thank you all in advance!
> > >
> > > Joe
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message