airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jack (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-3146) TimeDeltaSensor - consider also end time stamp
Date Wed, 03 Oct 2018 09:46:00 GMT
jack created AIRFLOW-3146:
-----------------------------

             Summary: TimeDeltaSensor  - consider also end time stamp
                 Key: AIRFLOW-3146
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3146
             Project: Apache Airflow
          Issue Type: Wish
            Reporter: jack


Currently the TimeDeltaSensor  takes these arguments:

 

{color:#4070a0}    :param delta: time length to wait after execution_date before succeeding{color}{color:#404040}
{color}{color:#4070a0}    :type delta: datetime.timedelta{color}{color:#404040}
{color}{color:#4070a0}    """{color}{color:#404040}

    {color}{color:#555555}@apply_defaults{color}{color:#404040}
    {color}{color:#007020}def{color} {color:#06287e}__init__{color}{color:#404040}({color}{color:#007020}self{color}{color:#404040},{color}
{color:#404040}delta{color}{color:#404040},{color} {color:#666666}*{color}{color:#404040}args{color}{color:#404040},{color}
{color:#666666}**{color}{color:#404040}kwargs{color}{color:#404040}):{color}{color:#404040}
        {color}{color:#007020}super{color}{color:#404040}({color}{color:#404040}TimeDeltaSensor{color}{color:#404040},{color}
{color:#007020}self{color}{color:#404040}){color}{color:#666666}.{color}{color:#06287e}__init__{color}{color:#404040}({color}{color:#666666}*{color}{color:#404040}args{color}{color:#404040},{color}
{color:#666666}**{color}{color:#404040}kwargs{color}{color:#404040}){color}{color:#404040}
        {color}{color:#007020}self{color}{color:#666666}.{color}{color:#404040}delta{color}
{color:#666666}={color} {color:#404040}delta{color}

 

*The problem:*

The operator assumes that the user would like to wait execution_date + delta.

This isn't always the case, The user might want to wait delta after the task has finished
running.

 

Lets take this example:

{color:#4070a0}    2016-01-01 can only start running on 2016-01-02. The timedelta here{color}{color:#404040}
{color}{color:#4070a0}    represents the time after the execution period has closed.{color}

Now, say the task ran for +4 hours.+ So if we set
{code:java}
delta = 5 minutes {code}
it actually doesn't wait at all. The sensor is useless in that senario.  What the user may
be intended is :

 

execution_date + duration to complete task + delta.

 

Or in simple words: Timestamp of execution end + delta.

 

*My suggestion:*


Add another Boolean  parameter which will choose if the delta is from execution_date or execution_date
+ duration of task



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message