airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-2138) Google Cloud Storage to Google Cloud Storage Operator to allow wildcards
Date Fri, 09 Mar 2018 21:23:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16393611#comment-16393611
] 

ASF subversion and git services commented on AIRFLOW-2138:
----------------------------------------------------------

Commit 803767959e27ea2f8424fd5afa40f1bab9f15648 in incubator-airflow's branch refs/heads/master
from [~danielkl3]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=8037679 ]

[AIRFLOW-2184][AIRFLOW-2138] Google Cloud Storage allow wildcards

- closes #2184

- Add support for wildcards to be provided in
source object argument
This allows the user of the Operator to provide a
wildcard in the format accepted by the
documentation. This message is echoed in the
docstring for ease of use, and also because it is
only three sentences and adding a link is not
required.
- Add an argument move_object (bool) to the
operator that, when true runs a mv command as
opposed to a cp command. That is to say, it moves
an object instead of copying the object. This is
especially useful when this operator is used to
move objects in the same bucket, perhaps from
folder to folder.
- Add dotmodus and dannylee12 to companies using
airflow
We use airflow in almost all of our projects.

-Unit tests written for the 3 use cases of the
added operator.

Remove newline

Split too long line over 2 lines.

Closes #3067 from DannyLee12/master


> Google Cloud Storage to Google Cloud Storage Operator to allow wildcards
> ------------------------------------------------------------------------
>
>                 Key: AIRFLOW-2138
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2138
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: contrib, gcp, operators
>    Affects Versions: 1.9.1
>            Reporter: Daniel
>            Assignee: Daniel
>            Priority: Minor
>             Fix For: 2.0.0
>
>   Original Estimate: 8h
>  Remaining Estimate: 8h
>
> Similar to AIRFLOW-2137 - Google Cloud Storage to BigQuery Operator to allow wildcards,
> This current operator requires the exact name of the object is provided as an argument.
> This issue differs slightly to the above issue as the hook that this operator is based
off of (GoogleCloudStorageHook) does not support wildcard expansion. Therefore, this can be
achieved by listing the objects that match the wildcards using either prefix=, suffix=, or
a combination of the two based on the position of the asterix in the string.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message