airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Martin (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (AIRFLOW-137) Airflow does not respect 'max_active_runs' when task from multiple dag runs cleared
Date Sat, 06 Jan 2018 00:07:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314155#comment-16314155
] 

Matt Martin edited comment on AIRFLOW-137 at 1/6/18 12:06 AM:
--------------------------------------------------------------

We are also running into this issue with Airflow 1.8.1.  My team helps to manage the airflow
instances at our company and one of the most frequent questions we field from users is why
the max_active_runs parameter is not respected when they backfill.  Folks generally expect
the behavior that [~mremes] described [here|https://issues.apache.org/jira/browse/AIRFLOW-137?focusedCommentId=16026063&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16026063]:

{quote}
I think the expected behaviour should be to schedule new dag runs till the limit of max_active_runs
is reached and poll the count continuously and add a new dag run when the active count is
less than the max value.
{quote}

I think AIRFLOW-1349 is probably relevant here.  Not sure the change that was done there really
makes sense to the average user (i.e. folks don't expect backfill to be exempted).  Having
the backfill wait and start as "capacity" becomes available seems like what people expect
(at least in our experience), but this approach is obviously more difficult/involved to implement
than the two earlier options (i.e. ignoring the limit or failing fast).


was (Author: matt.martin):
We are also running into this issue with Airflow 1.8.0.  My team helps to manage the airflow
instances at our company and one of the most frequent questions we field from users is why
the max_active_runs parameter is not respected when they backfill.  Folks generally expect
the behavior that [~mremes] described [here|https://issues.apache.org/jira/browse/AIRFLOW-137?focusedCommentId=16026063&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16026063]:

{quote}
I think the expected behaviour should be to schedule new dag runs till the limit of max_active_runs
is reached and poll the count continuously and add a new dag run when the active count is
less than the max value.
{quote}

I think AIRFLOW-1349 is probably relevant here.  Not sure the change that was done there really
makes sense to the average user (i.e. folks don't expect backfill to be exempted).  Having
the backfill wait and start as "capacity" becomes available seems like what people expect
(at least in our experience), but this approach is obviously more difficult/involved to implement
than the two earlier options (i.e. ignoring the limit or failing fast).

> Airflow does not respect 'max_active_runs' when task from multiple dag runs cleared
> -----------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-137
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-137
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Tomasz Bartczak
>            Priority: Minor
>
> Also requested at https://github.com/apache/incubator-airflow/issues/1442
> Dear Airflow Maintainers,
> Environment
> Before I tell you about my issue, let me describe my Airflow environment:
> Please fill out any appropriate fields:
>     Airflow version: 1.7.0
>     Airflow components: webserver, mysql, scheduler with celery executor
>     Python Version: 2.7.6
>     Operating System: Linux Ubuntu 3.19.0-26-generic Scheduler runs with --num-runs and
get restarted around every minute or so
> Description of Issue
> Now that you know a little about me, let me tell you about the issue I am having:
>     What did you expect to happen?
>     After running 'airflow clear -t spark_final_observations2csv -s 2016-04-07T01:00:00
-e 2016-04-11T01:00:00 MODELLING_V6' I expected that this task gets executed in all dag-runs
in specified by given time-range - respecting 'max_active_runs'
>     Dag configuration:
>     concurrency= 3,
>     max_active_runs = 2,
>     What happened instead?
>     Airflow at first started executing 3 of those tasks, which already violates 'max_active_runs',
but it looks like 'concurrency' was the applied limit here.
>     3_running_2_pending
> After first task was done - airflow scheduled all other tasks, making it 5 running dags
at the same time that violates all specified limit.
> In the GUI we saw red warning (5/2 Dags running ;-) )
> Reproducing the Issue
> max_active_runs is respected in a day-to-day basis - when of the tasks was stuck - airflow
didn't start more than 2 dags concurrently.
> [screenshots in the original issue: https://github.com/apache/incubator-airflow/issues/1442]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message