airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jerry Chi <jerry....@smartnews.com>
Subject Re: Guidelines around how to scale with worker nodes?
Date Mon, 29 Oct 2018 00:41:33 GMT
Sorry, any tips or hints related the below questions? Thank you.

Jerry

2018年10月24日(水) 3:37、Jerry Chi さん(jerry.chi@smartnews.com)のメッセージ:

> Hi everyone,
>
> I'm occasionally observing tasks stuck in "queued" for a long time despite
> trying various edits of parameter values in airflow.cfg and I'm guessing it
> would help to increase the number of worker nodes (right now I have one
> worker node).
>
> Are there any guidelines for:
> 1. How to determine if the # of worker nodes is indeed the bottleneck
> causing tasks to be stuck in "queued" ? It doesn't seem the memory/CPU
> usage on the worker node is close to 100%.
> 2. How to determine the optimal number and CPU/memory specs of the worker
> nodes if I want to be able to handle X simultaneous tasks without them
> getting stuck in "queued" ?
> I'm using CeleryExecutor + RabbitMQ on EC2.
>
> Thanks~
> Jerry Chi ジェリー・チー | Data Science Manager | +81-70-2668-5491 | LINE/Skype:
> peacej | 카톡: peacej2 | WeChat: jerrychijerry
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message