airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alberto Anceschi (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-3244) Introduce offset on the execution date for data assessment
Date Tue, 23 Oct 2018 10:20:00 GMT
Alberto Anceschi created AIRFLOW-3244:
-----------------------------------------

             Summary: Introduce offset on the execution date for data assessment
                 Key: AIRFLOW-3244
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3244
             Project: Apache Airflow
          Issue Type: Improvement
          Components: DAG
    Affects Versions: 1.10.0
            Reporter: Alberto Anceschi


Hi everyone,

 

I'm trying to port my current cronjobs into Airflow. Let's consider a real case scenario:
I've to send every week a report and through the pipeline data from Google Analytics needs
to be collected, so I need 2 days before running the DAG (data assessment). Week starts on
Monday and ends on Sunday, so I need the DAG to run on Wednesday at Midnight UTC.

In order to see on the Airflow dashboard start_date/exection_date that make sense to me, for
now I've used a TimeDeltaSensor that adds that 2 day offset I need, but this is not its purpose.
I also use Celery executor, so its workers keep polling during those 2 days, making them
unavailable for other DAGs.

I think that the assumption that at the end of the period scheduled data are ready is not
correct and at the same time it's much more intuitive seeing on the dashboard Monday execution
dates instead of Tuesday ones.

 

What do you think about this request? Any suggestion? Thank you,

 

Alberto

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message