airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ash Berlin-Taylor (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-3244) Introduce offset on the execution date for data assessment
Date Tue, 23 Oct 2018 10:27:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660409#comment-16660409
] 

Ash Berlin-Taylor commented on AIRFLOW-3244:
--------------------------------------------

Not commenting on the whole feature request: but:

> I also use Celery executor, so its workers keep polling during those 2 days, making them
unavailable for other DAGs.

In the next release Sensors can be configured to check once then release the executor slot
back which would address this one point.

> Introduce offset on the execution date for data assessment
> ----------------------------------------------------------
>
>                 Key: AIRFLOW-3244
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3244
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: DAG
>    Affects Versions: 1.10.0
>            Reporter: Alberto Anceschi
>            Priority: Minor
>              Labels: features, request
>
> Hi everyone,
>  
> I'm trying to port my current cronjobs into Airflow. Let's consider a real case scenario:
I've to send every week a report and through the pipeline data from Google Analytics needs
to be collected, so I need 2 days before running the DAG (data assessment). Week starts on
Monday and ends on Sunday, so I need the DAG to run on Wednesday at Midnight UTC.
> In order to see on the Airflow dashboard start_date/exection_date that make sense to
me, for now I've used a TimeDeltaSensor that adds that 2 day offset I need, but this is not
its purpose. I also use Celery executor, so its workers keep polling during those 2 days,
making them unavailable for other DAGs.
> I think that the assumption that at the end of the period scheduled data are ready is
not correct and at the same time it's much more intuitive seeing on the dashboard Monday execution
dates instead of Tuesday ones.
>  
> What do you think about this request? Any suggestion? Thank you,
>  
> Alberto
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message