airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yannick Einsweiler (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-1894) Rebase and migrate existing Airflow GCP operators to google-python-cloud
Date Tue, 17 Apr 2018 21:13:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441510#comment-16441510
] 

Yannick Einsweiler commented on AIRFLOW-1894:
---------------------------------------------

[~wileeam] read my initial comment, the idea was to leverage [https://cloud.google.com/storage/transfer/reference/rest/]
*without* the need for a local or virtual machine passthrough.

We ended writing that operator (and a sensor that checks when job is complete), all triggered
from within Gcloud but have since then abandoned the project and have our AWS Lambda save
output to GCS directly.

> Rebase and migrate existing Airflow GCP operators to google-python-cloud
> ------------------------------------------------------------------------
>
>                 Key: AIRFLOW-1894
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1894
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: contrib
>    Affects Versions: Airflow 2.0
>            Reporter: Feng Lu
>            Assignee: Feng Lu
>            Priority: Minor
>
> [google-api-python-client|https://github.com/google/google-api-python-client] is in maintenance
mode and it's recommended that [google-cloud-python|https://github.com/GoogleCloudPlatform/google-cloud-python]
should be used whenever possible. Given that we don't have feature parity between the two
libraries, this issue is created to track the long-term migration efforts moving from google-api-python-client
to google-cloud-python. Here are some general guidelines we try to follow in this cleanup
process:
> - add google-cloud-python dependency as part of gcp_api extra packages (make sure there
is no dependency conflict between the two).
> - new operators shall be based on google-cloud-python if possible.
> - migrate existing GCP operators when the underlying GCP service is available in google-cloud-python.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message