airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shenghu Yang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-111) DAG concurrency is not honored
Date Fri, 13 May 2016 20:25:12 GMT
Shenghu Yang created AIRFLOW-111:
------------------------------------

             Summary: DAG concurrency is not honored
                 Key: AIRFLOW-111
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-111
             Project: Apache Airflow
          Issue Type: Sub-task
          Components: celery, scheduler
    Affects Versions: Airflow 1.6.2
         Environment: Version of Airflow: 1.6.2
Airflow configuration: Running a Scheduler with LocalExecutor
Operating System: 3.13.0-74-generic #118-Ubuntu SMP Thu Dec 17 22:52:10 UTC 2015 x86_64 x86_64
x86_64 GNU/Linux
Python Version: 2.7.6
Screen shots of your DAG's status:
            Reporter: Shenghu Yang


Description of Issue

In our dag, we set the dag_args['concurrency'] = 8, however, when the scheduler starts to
run, we can see this concurrency is not being honored, airflow scheduler will run up to num
of the 'parallelism' (we set as 25) jobs.

What did you expect to happen?
dag_args['concurrency'] = 8 is honored, e.g. only run at most 8 jobs concurrently.

What happened instead?
when the dag starts to run, we can see the concurrency is not being honored, airflow scheduler/celery
worker will run up to the 'parallelism' (we set as 25) jobs.

Here is how you can reproduce this issue on your machine:

create a dag which contains nothing but 25 parallelized jobs.
set the dag dag_args['concurrency'] = 8
set the airflow parallelism to 25
then run: airflow scheduler
you will see all 25 jobs are scheduled to run, not 8.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message