airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kevin Yuen (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-513) ExternalTaskSensor tasks should not count towards parallelism limit
Date Fri, 16 Sep 2016 13:39:22 GMT
Kevin Yuen created AIRFLOW-513:
----------------------------------

             Summary: ExternalTaskSensor tasks should not count towards parallelism limit
                 Key: AIRFLOW-513
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-513
             Project: Apache Airflow
          Issue Type: Improvement
         Environment: Ubuntu 14.04
Version 1.7.0

            Reporter: Kevin Yuen


Hi, 

We are using airflow version 1.7.0 and we are using `ExternalTaskSensor` pretty heavily to
manage dependencies between our DAGs. 

We have recently experienced a case where the external task sensors are causing the DAGs to
go into limbo state because they took up all the execution slots defined via `AIRFLOW__CORE__PARALLELISM`.


For example: 
Given we have 2 DAGs: 
first one with 16 python operator tasks, and the other with 16 sensors. We set `PARALLELISM`
to 16. 

If the scheduler choses to schedule all 16 sensors first, the dag runs will never complete.


There are a couple of work around to this:
#. staggering the DAGs so that the first dag with python operator runs first
#. lowering the TaskSensor timeout thresholds and relying on retries

Both of these options seems unideal to us and we wonder if `ExternalTaskSensor` should really
be counting towards the `PARALLELISM` limit?

Cheers, 
Kevin





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message