airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pramiti Goel <>
Subject Celery Accepting more tasks than worker concurrency
Date Fri, 26 Apr 2019 05:01:35 GMT
I am running few long running tasks in dedicated queue. So there are no
short time running tasks.  Currently when i start many multiple tasks on
that queue, the new tasks are getting submitted to the worker which is
already full i.e no of tasks = worker concurrency. The worker 2 listening
to same queue is still has less tasks and can accept more tasks. Ideally
the task should go to worker 2.

What is the reason for this behaviour ? Is it because of prefetching ?

We further see that the task which got submitted to worker 1 which is
already full, goes to worker 2 after sometime i.e around 2 hours. Why does
this happen ? And why after 2 hours ? Is there a way to reduce the time the
task goes from worker 1 to worker 2.

My late acknowledgement is set = true
and visibility timeout on worker = 6hrs.
broker is redis.
celery version 4.2.1

I have attached few screenshot to catch this behaviour:
My other worker is still empty, but it goes to the worker which is already

I could go close to this only in this link
Please check trailblazer.streaming.growth_fortune
trailblazer-growth-fortune-stream-0001 2019-04-25T10:10:00 as an example.
Screenshot 1: Airflow Ui
Screenshot 2: Celery Flower
[image: Screen Shot 2019-04-26 at 12.02.47 PM.png]

View raw message