airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tomasz Bartczak (JIRA)" <>
Subject [jira] [Created] (AIRFLOW-137) Airflow does not respect 'max_active_runs' when task from multiple dag runs cleared
Date Thu, 19 May 2016 09:04:13 GMT
Tomasz Bartczak created AIRFLOW-137:

             Summary: Airflow does not respect 'max_active_runs' when task from multiple dag
runs cleared
                 Key: AIRFLOW-137
             Project: Apache Airflow
          Issue Type: Bug
            Reporter: Tomasz Bartczak
            Priority: Minor

Also requested at

Dear Airflow Maintainers,

Before I tell you about my issue, let me describe my Airflow environment:

Please fill out any appropriate fields:

    Airflow version: 1.7.0
    Airflow components: webserver, mysql, scheduler with celery executor
    Python Version: 2.7.6
    Operating System: Linux Ubuntu 3.19.0-26-generic Scheduler runs with --num-runs and get
restarted around every minute or so

Description of Issue

Now that you know a little about me, let me tell you about the issue I am having:

    What did you expect to happen?
    After running 'airflow clear -t spark_final_observations2csv -s 2016-04-07T01:00:00 -e
2016-04-11T01:00:00 MODELLING_V6' I expected that this task gets executed in all dag-runs
in specified by given time-range - respecting 'max_active_runs'
    Dag configuration:
    concurrency= 3,
    max_active_runs = 2,

    What happened instead?
    Airflow at first started executing 3 of those tasks, which already violates 'max_active_runs',
but it looks like 'concurrency' was the applied limit here.

After first task was done - airflow scheduled all other tasks, making it 5 running dags at
the same time that violates all specified limit.
In the GUI we saw red warning (5/2 Dags running ;-) )

Reproducing the Issue

max_active_runs is respected in a day-to-day basis - when of the tasks was stuck - airflow
didn't start more than 2 dags concurrently.

[screenshots in the original issue:]

This message was sent by Atlassian JIRA

View raw message