airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Lamblin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-363) Celery Queue becoming too huge when worker unavailable
Date Wed, 18 Apr 2018 09:14:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16442170#comment-16442170
] 

Daniel Lamblin commented on AIRFLOW-363:
----------------------------------------

I'm not sure, but I think #3 was done, right [~criccomini]?

> Celery Queue becoming too huge when worker unavailable
> ------------------------------------------------------
>
>                 Key: AIRFLOW-363
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-363
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: celery
>    Affects Versions: Airflow 1.7.1.3
>            Reporter: Vineet Goel
>            Priority: Major
>
> The scheduler currently queues executable tasks every run of the scheduler. I ran into
an issue where I was using a single worker and my dag was composed of a bunch of tasks that
could run simultaneously. I was setting a concurrency on the worker to control how many of
those tasks could run at the same time. However, my scheduler kept queueing more and more
tasks. This can cause a HUGE back pressure on the celery queue, which becomes unmanageable
at times. 
> We should set the http://docs.celeryproject.org/en/latest/configuration.html#celery-event-queue-ttl
> configuration to handle this. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message