airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sai Phanindhra <phani8...@gmail.com>
Subject Re: run tasks parallel in airflow
Date Tue, 11 Dec 2018 04:10:05 GMT
Hi,
     Along with celery worker, you might have to configure dag concurrency
and number of celery workers to get maximum parallelism in airflow. As long
as scheduler has enough resources to schedule tasks and enough workers to
run the job, you can parallely run a lot of tasks in airflow.

On Tue 11 Dec, 2018, 08:32 lkeswar@gmail.com <lkeswar@gmail.com wrote:

> Hi,
>
>    Can you please guide us how to run the tasks parallely in airflow as
> which i need to run almost 14 tasks parallely. In config file executor as
> 'celery'. Is there any thing which i need to concentrate or guide me to run
> the tasks.
>
>
> thx,
> Lokesh
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message