airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kaxil Naik (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-1855) Add an Operator to copy files with a specific delimiter in a directory from one GCS bucket to another
Date Tue, 28 Nov 2017 10:36:00 GMT
Kaxil Naik created AIRFLOW-1855:
-----------------------------------

             Summary: Add an Operator to copy files with a specific delimiter in a directory
from one GCS bucket to another
                 Key: AIRFLOW-1855
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1855
             Project: Apache Airflow
          Issue Type: New Feature
          Components: contrib, gcp
    Affects Versions: 1.10.0
            Reporter: Kaxil Naik
            Assignee: Kaxil Naik
            Priority: Minor
             Fix For: 1.10.0


Use case: Copy all the CSV/JSON files from a particular directory in a Bucket to another bucket
and in a specific directory (or the same).

Proposed Approach:
- Add 'delimiter' argument in GCP hook to filter files with a particular delimiter.
- Get the list of files to copy and filter it with delimiter using 'list' method in GCP hook
- Use loop and 'copy' method in GCP hook. 


Note: Under the hood GCS has no directories. Files are just objects.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message