airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guillermo Rodríguez Cano (JIRA) <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-1894) Rebase and migrate existing Airflow GCP operators to google-python-cloud
Date Tue, 17 Apr 2018 21:38:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441527#comment-16441527
] 

Guillermo Rodríguez Cano commented on AIRFLOW-1894:
---------------------------------------------------

I did read your comment, [~yiga2], and I did not say anything about that in my comment (In
fact I think that what you did could be a complement to the standard google-cloud-python library
as I haven't checked it enough to conclude whether it is possible to stream a file or not).

I assume the google-cloud-python library is better performing than the currently used one
but the changes required in hooks are quite some and so I was requesting for more information
on that, and offered my help to change it (and try to figure out if it is possible to emulate
such storage transfer service you point and that I guess it is the same as the transfer service
offered in the UI of Google Cloud Console).

> Rebase and migrate existing Airflow GCP operators to google-python-cloud
> ------------------------------------------------------------------------
>
>                 Key: AIRFLOW-1894
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1894
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: contrib
>    Affects Versions: Airflow 2.0
>            Reporter: Feng Lu
>            Assignee: Feng Lu
>            Priority: Minor
>
> [google-api-python-client|https://github.com/google/google-api-python-client] is in maintenance
mode and it's recommended that [google-cloud-python|https://github.com/GoogleCloudPlatform/google-cloud-python]
should be used whenever possible. Given that we don't have feature parity between the two
libraries, this issue is created to track the long-term migration efforts moving from google-api-python-client
to google-cloud-python. Here are some general guidelines we try to follow in this cleanup
process:
> - add google-cloud-python dependency as part of gcp_api extra packages (make sure there
is no dependency conflict between the two).
> - new operators shall be based on google-cloud-python if possible.
> - migrate existing GCP operators when the underlying GCP service is available in google-cloud-python.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message