airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ash Berlin-Taylor <...@apache.org>
Subject Re: [DISCUSS] AIP-5: Remote DAG Fetcher
Date Thu, 02 May 2019 09:31:11 GMT
I've left some comments on the wiki page. In short: nice idea, but I'd like to see a few more
of the ideas expanded upon first before we vote on this.

-ash

> On 1 May 2019, at 05:23, Chao-Han Tsai <milton0825@gmail.com> wrote:
> 
> Hi all,
> 
> I would like to bump this thread up again to seek for more feedbacks from
> you for the AIP-5:
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher
> 
> In short, this AIP aims to enable Airflow to fetch DAG files from remote
> storage systems (git, s3,...) outside the file system local to the service.
> This grant a much greater flexibility, eases implementation, and
> standardizes ways to sync remote sources of DAGs with Airflow. It also
> opens up new opportunities for Airflow such as exposing all versions of
> DAGs to Airflow to support versioned DagRuns.
> 
> Cheers,
> 
> Chao-Han
> 
> 
> 
> On Sat, Mar 23, 2019 at 9:56 PM Chao-Han Tsai <milton0825@gmail.com> wrote:
> 
>> Hi Airflowers,
>> 
>> We had an email thread
>> <https://lists.apache.org/thread.html/03ddcd3a42b7fd6e3dad9711e8adea37fc00391f6053762f73af5b6a@%3Cdev.airflow.apache.org%3E>
>> and PR <https://github.com/apache/airflow/pull/3138> about supporting
>> remote DAG fetcher in the past and there was an AIP
>> <https://cwiki.apache.org/confluence/display/AIRFLOW/Deprecated-+AIP-5+DagFetcher>
about
>> it. But the author seems to be not working on it anymore, so I picked it up
>> and came up with a new AIP:
>> 
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher
>> 
>> I would like to hear your feedback and suggestions for improvement!
>> 
>> Chao-Han Tsai
>> 
>> 
> 
> -- 
> 
> Chao-Han Tsai


Mime
View raw message