airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [airflow] feluelle commented on issue #6428: [AIRFLOW-5753] Add DailyLatestOnlyOperator
Date Sat, 26 Oct 2019 14:16:09 GMT
feluelle commented on issue #6428: [AIRFLOW-5753] Add DailyLatestOnlyOperator
URL: https://github.com/apache/airflow/pull/6428#issuecomment-546606824
 
 
   > This is useful when we scrape external data from third-parties on an hourly basis/for
multiple times in a day to keep data up-to-date. If the DAG is paused for several days, we
want to avoid multiple runs in a day, but still want to ensure data is filled in every daily
window.
   
   We (at work) have a similiar case where we are always requesting data from the last 3 months.

   Our first task is a `ShortCircuitOperator` that checks if the current run really needs
to run i.e. the execution date is in a list of dates or it is the latest execution date.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message