Mailing-List: contact commits-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
Date: Thu, 28 Jul 2016 22:15:20 +0000 (UTC)
From: "Vineet Goel (JIRA)" <jira@apache.org>
To: commits@airflow.incubator.apache.org
Message-ID: <JIRA.12992436.1469490152000.162519.1469744120990@Atlassian.JIRA>
In-Reply-To: <JIRA.12992436.1469490152000@Atlassian.JIRA>
References: <JIRA.12992436.1469490152000@Atlassian.JIRA> <JIRA.12992436.1469490152085@arcas>
Subject: [jira] [Commented] (AIRFLOW-363) Celery Queue becoming too huge
 when worker unavailable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 28 Jul 2016 22:15:26 -0000


    [ https://issues.apache.org/jira/browse/AIRFLOW-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15398302#comment-15398302 ] 

Vineet Goel commented on AIRFLOW-363:
-------------------------------------

[~criccomini] Its good to know that periodic scheduler restarts aren't required anymore :)

> Celery Queue becoming too huge when worker unavailable
> ------------------------------------------------------
>
>                 Key: AIRFLOW-363
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-363
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: celery
>    Affects Versions: Airflow 1.7.1.3
>            Reporter: Vineet Goel
>
> The scheduler currently queues executable tasks every run of the scheduler. I ran into an issue where I was using a single worker and my dag was composed of a bunch of tasks that could run simultaneously. I was setting a concurrency on the worker to control how many of those tasks could run at the same time. However, my scheduler kept queueing more and more tasks. This can cause a HUGE back pressure on the celery queue, which becomes unmanageable at times. 
> We should set the http://docs.celeryproject.org/en/latest/configuration.html#celery-event-queue-ttl
> configuration to handle this. 


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)