airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vineet Goel (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-363) Celery Queue becoming too huge when worker unavailable
Date Mon, 25 Jul 2016 23:44:20 GMT


Vineet Goel commented on AIRFLOW-363:


> Celery Queue becoming too huge when worker unavailable
> ------------------------------------------------------
>                 Key: AIRFLOW-363
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Vineet Goel
> The scheduler currently queues executable tasks every run of the scheduler. I ran into
an issue where I was using a single worker and my dag was composed of a bunch of tasks that
could run simultaneously. I was setting a concurrency on the worker to control how many of
those tasks could run at the same time. However, my scheduler kept queueing more and more
tasks. This can cause a HUGE back pressure on the celery queue, which becomes unmanageable
at times. 
> We should set the
> configuration to handle this. 

This message was sent by Atlassian JIRA

View raw message