airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Riccomini (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-113) DAG concurrency is not honored
Date Fri, 13 May 2016 23:00:15 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15283273#comment-15283273
] 

Chris Riccomini commented on AIRFLOW-113:
-----------------------------------------

Also, I wonder if perhaps you want max_active_runs instead of concurrency?

>From [FAQ|https://pythonhosted.org/airflow/faq.html]: 

{quote}
concurency defines how many running task instances a DAG is allowed to have, beyond which
point things get queued
{quote}

Whereas:

{quote}
Is the max_active_runs parameter of your DAG reached? max_active_runs defines how many running
concurrent instances of a DAG there are allowed to be.
{quote}

> DAG concurrency is not honored
> ------------------------------
>
>                 Key: AIRFLOW-113
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-113
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: celery, executor, scheduler
>    Affects Versions: Airflow 1.6.2
>         Environment: Version of Airflow: 1.6.2
> Airflow configuration: Running a Scheduler with LocalExecutor or CeleryExecutor
> Operating System: 3.13.0-74-generic #118-Ubuntu SMP Thu Dec 17 22:52:10 UTC 2015 x86_64
x86_64 x86_64 GNU/Linux
> Python Version: 2.7.6
>            Reporter: Shenghu Yang
>
> This is reported in airflow github also: https://github.com/apache/incubator-airflow/issues/1424
> In our dag, we set the dag_args['concurrency'] = 8, however, when the scheduler starts
to run, we can see this concurrency is not being honored, airflow scheduler will run up to
num of the 'parallelism' (we set as 25) jobs.
> What did you expect to happen?
> dag_args['concurrency'] = 8 is honored, e.g. only run at most 8 jobs concurrently.
> What happened instead?
> when the dag starts to run, we can see the concurrency is not being honored, airflow
scheduler will run up to the 'parallelism' (we set as 25) jobs.
> Here is how you can reproduce this issue on your machine:
> create a dag which contains nothing but 25 parallelized jobs.
> set the dag dag_args['concurrency'] = 8
> set the airflow parallelism to 25
> then run: airflow scheduler
> you will see all 25 jobs are scheduled to run, not 8.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message