airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Angela Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-1689) Airflow doesn't schedule tasks if there are too many failed tasks in a previous dag run
Date Fri, 06 Oct 2017 15:42:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-1689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Angela Zhang updated AIRFLOW-1689:
----------------------------------
    Description: 
see: !https://d3uik7083b30dy.cloudfront.net/items/043p0z0h2h0J42393E3i/Image%202017-10-06%20at%2011.26.35%20AM.png?X-CloudApp-Visitor-Id=1681354&v=3020d646!

This happens to multiple of our dags, where if previous dag runs of this dag have too many
failed / running tasks (even though it's still lower than the concurrency / max active dags
runs,  concurrency=16, max_active_runs=16), the scheduler does not schedule any more tasks
in the latest dag run for this dag. 

!https://d3uik7083b30dy.cloudfront.net/items/0r3r2B141M233Z2r0C1B/Image%202017-10-06%20at%2011.38.17%20AM.png?X-CloudApp-Visitor-Id=1681354&v=1903b2e6!

We can make the scheduler schedule more tasks if we mark the previous dag runs as successful,
but this still seems like a bug in airflow. 


  was:
see: !https://cl.ly/0I272J1N1N0m!

This happens to multiple of our dags, where if previous dag runs of this dag have too many
failed / running tasks (even though it's still lower than the concurrency / max active dags
runs,  concurrency=16, max_active_runs=16), the scheduler does not schedule any more tasks
in the latest dag run for this dag. 

!https://cl.ly/3g1E1e2h0634!

We can make the scheduler schedule more tasks if we mark the previous dag runs as successful,
but this still seems like a bug in airflow. 



> Airflow doesn't schedule tasks if there are too many failed tasks in a previous dag run
> ---------------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-1689
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1689
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Angela Zhang
>
> see: !https://d3uik7083b30dy.cloudfront.net/items/043p0z0h2h0J42393E3i/Image%202017-10-06%20at%2011.26.35%20AM.png?X-CloudApp-Visitor-Id=1681354&v=3020d646!
> This happens to multiple of our dags, where if previous dag runs of this dag have too
many failed / running tasks (even though it's still lower than the concurrency / max active
dags runs,  concurrency=16, max_active_runs=16), the scheduler does not schedule any more
tasks in the latest dag run for this dag. 
> !https://d3uik7083b30dy.cloudfront.net/items/0r3r2B141M233Z2r0C1B/Image%202017-10-06%20at%2011.38.17%20AM.png?X-CloudApp-Visitor-Id=1681354&v=1903b2e6!
> We can make the scheduler schedule more tasks if we mark the previous dag runs as successful,
but this still seems like a bug in airflow. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message