airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Diogo Franco (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-2221) Fill up DagBag from remote locations
Date Sun, 18 Mar 2018 01:31:00 GMT


Diogo Franco commented on AIRFLOW-2221:

[PR for this issue|]

> Fill up DagBag from remote locations
> ------------------------------------
>                 Key: AIRFLOW-2221
>                 URL:
>             Project: Apache Airflow
>          Issue Type: New Feature
>          Components: configuration, core
>    Affects Versions: Airflow 2.0
>            Reporter: Diogo Franco
>            Assignee: Diogo Franco
>            Priority: Major
>             Fix For: Airflow 2.0
> The ability to fill up the DagBag from remote locations (HDFS, S3...) seems to be deemed
useful, e.g. facilitating deployment processes.
> This JIRA is to propose an implementation of a *DagFetcher* abstraction on the DagBag,
where the collect_dags method can delegate the walking to a *FileSystemDagFetcher*, *GitRepoDagFetcher*,
*S3DagFetcher*, *HDFSDagFetcher*, *GCSDagFetcher*, *ArtifactoryDagFetcher* or even *TarballInS3DagFetcher*.
> This was briefly discussed in [this mailing list thread|]
> I'm happy to start work on this and provide an initial implementation for review.

This message was sent by Atlassian JIRA

View raw message