beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-1648) Replace gsutil calls with Cloud Storage API
Date Tue, 28 Mar 2017 22:15:41 GMT

    [ https://issues.apache.org/jira/browse/BEAM-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946067#comment-15946067
] 

ASF GitHub Bot commented on BEAM-1648:
--------------------------------------

Github user asfgit closed the pull request at:

    https://github.com/apache/beam/pull/2333


> Replace gsutil calls with Cloud Storage API
> -------------------------------------------
>
>                 Key: BEAM-1648
>                 URL: https://issues.apache.org/jira/browse/BEAM-1648
>             Project: Beam
>          Issue Type: Wish
>          Components: sdk-py
>    Affects Versions: 0.5.0
>            Reporter: David Volquartz Lebech
>            Assignee: Ahmet Altay
>
> When using the {{DataflowRunner}} and {{--setup-file}} parameter, {{gsutil}} is used
for _some_ of the Cloud Storage uploads [here|https://github.com/apache/beam/blob/466599d765aa82acaf997ec8776405152bbde4c1/sdks/python/apache_beam/runners/dataflow/internal/dependency.py#L89-L90].
This makes it difficult to run a pipeline in an environment where the Cloud Platform tools
are not installed -- e.g. a Docker Python container or a Heroku instance.
> The Storage API is used in other places such as [here|https://github.com/apache/beam/blob/466599d765aa82acaf997ec8776405152bbde4c1/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py#L431-L432]
when staging the session and main SDK package, so I'm unsure if this is by design or an inconsistency
in the usage.
> Thank you for considering this.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message