airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josef Samanek"<josef.sama...@kiwi.com>
Subject Re: Tasks Queued but never run
Date Mon, 05 Jun 2017 14:12:11 GMT
Hey. Thanks for the answer. I previously also tried to run scheduler -n 10, but it was back
when I was still using LocalExecutor. And it did not help. I have not yet tried to do it with
CeleryExecutor, so I might. 

Still, I would prefer to find an actual solution for the underlying problem, not just a workaround
(eventhough a working workaround is also appreciated).

Best regards,
Joe

On 2017-06-02 00:10 (+0200), Alex Guziel <alex.guziel@airbnb.com.INVALID> wrote: 
> We've noticed this with celery, relating to this
> https://github.com/celery/celery/issues/3765
> 
> We also use `-n 5` option on the scheduler so it restarts every 5 runs,
> which will reset all queued tasks.
> 
> Best,
> Alex
> 
> On Thu, Jun 1, 2017 at 2:18 PM, Josef Samanek <josef.samanek@gmail.com>
> wrote:
> 
> > Hi!
> >
> > We have a problem with our airflow. Sometimes, several tasks get queued
> > but they never get run and remain in Queud state forever. Other tasks from
> > the same schedule interval run. And next schedule interval runs normally
> > too. But these several tasks remain queued.
> >
> > We are using Airflow 1.8.1. Currently with CeleryExecutor and redis, but
> > we had the same problem with LocalExecutor as well (actually switching to
> > Celery helped quite a bit, the problem now happens way less often, but
> > still it happens). We have 18 DAGs total, 13 active. Some have just 1-2
> > tasks, but some are more complex, like 8 tasks or so and with upstreams.
> > There are also ExternalTaskSensor tasks used.
> >
> > I tried playing around with DAG configurations (limiting concurrency,
> > max_active_runs, ...), tried switching off some DAGs completely (not all
> > but most) etc., so far nothing helped. Right now, I am not really sure,
> > what else to try to identify a solve the issue.
> >
> > I am getting a bit desperate, so I would really appreciate any help with
> > this. Thank you all in advance!
> >
> > Joe
> >
> 

Mime
View raw message