airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-3902) Singleton operator to skip downstream tasks
Date Wed, 11 Sep 2019 15:28:01 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927687#comment-16927687
] 

ASF GitHub Bot commented on AIRFLOW-3902:
-----------------------------------------

stale[bot] commented on pull request #4727: [AIRFLOW-3902] Add new singleton_operator and
unittest
URL: https://github.com/apache/airflow/pull/4727
 
 
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Singleton operator to skip downstream tasks
> -------------------------------------------
>
>                 Key: AIRFLOW-3902
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3902
>             Project: Apache Airflow
>          Issue Type: New Feature
>          Components: operators
>            Reporter: Kiran Pathak
>            Assignee: Kiran Pathak
>            Priority: Major
>              Labels: operator, singleton
>         Attachments: image-2019-02-18-05-03-00-290.png, image-2019-02-18-05-03-58-662.png
>
>
> This operator will:
>  * check for downstream tasks from previous DagRuns in *_unfinished_* state
>  * skip downstream tasks from execution if it finds a downstream task in running state
from previous DagRuns
> This feature ensures downstream children of this operator only one instance is scheduled
to run. It avoids unnecessary processing / scheduling if it does not need to.
>   !image-2019-02-18-05-03-00-290.png|width=213,height=102!
> !image-2019-02-18-05-03-58-662.png|width=224,height=88!
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Mime
View raw message