airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-2033) Create Google Cloud Storage List Operator
Date Fri, 26 Jan 2018 10:29:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16340883#comment-16340883
] 

ASF subversion and git services commented on AIRFLOW-2033:
----------------------------------------------------------

Commit f9ddb36df1676afea22125644c241c8ba65598db in incubator-airflow's branch refs/heads/master
from [~kaxilnaik]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=f9ddb36 ]

[AIRFLOW-2033] Add Google Cloud Storage List Operator

Added an operator to get object names in a GCS
bucket filtered by prefix and delimiter with
example.

Closes #2974 from kaxil/gcs_list_op


> Create Google Cloud Storage List Operator
> -----------------------------------------
>
>                 Key: AIRFLOW-2033
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2033
>             Project: Apache Airflow
>          Issue Type: Task
>          Components: gcp
>    Affects Versions: Airflow 2.0, 1.10.0
>            Reporter: Kaxil Naik
>            Assignee: Kaxil Naik
>            Priority: Trivial
>             Fix For: 2.0.0
>
>
> There is an option in GCS hook to get the names of objects in a bucket filtered by prefix
and delimiter. An operator that returns would be helpful. This operator would return a python
list containing object names. A downstream task with python operator can be used to retrieve
these file names using xcom.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message