airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Feng Lu (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-15) Remove GCloud from Airflow
Date Thu, 07 Dec 2017 16:12:07 GMT


Feng Lu commented on AIRFLOW-15:

[~criccomini] [~barrywhart]  Unfortunately we don't have feature parity between the two yet,
but it's expected that all GCP services will be supported in google-cloud-python. Checked
with our internal team, it's recommended that google-cloud-python should be used whenever
possible. Specifically for Airflow GCP connectors, I would propose:
- add google-cloud-python dependency as part of gcp_api.
- new operators shall base on google-cloud-python if possible.
- we (or anyone outside google) can migrate existing GCP operators when the underlying GCP
service is available in google-cloud-python. 

Feel free to assign this JIRA issue to me for process tracking. 

> Remove GCloud from Airflow
> --------------------------
>                 Key: AIRFLOW-15
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Task
>          Components: gcp
>            Reporter: Chris Riccomini
>            Assignee: Chris Riccomini
>              Labels: gcp
> After speaking with Google, there was some concern about using the [gcloud-python|]
library for Airflow. There are several concerns:
> # It's not clear (even to people at Google) what this library is, who owns it, etc.
> # It does not support all services (the way [google-api-python-client|]
> # There are compatibility issues between google-api-python-client and gcloudpython.
> We currently support both, after libraries depending on which package you you install:
{{airfow[gcp_api]}} or {{airflow[gcloud]}}. This ticket is to remove the {{airflow[gcloud]}}
packaged, and all associated code.
> The main associated code, afaik, is the use of the {{gcloud}} library in the Google cloud
storage hooks/operators--specifically for Google cloud storage Airfow logging.

This message was sent by Atlassian JIRA

View raw message