beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mike Lambert (JIRA)" <>
Subject [jira] [Comment Edited] (BEAM-1790) Failure to build --requirements.txt when it uses google protobuf
Date Thu, 23 Mar 2017 08:25:41 GMT


Mike Lambert edited comment on BEAM-1790 at 3/23/17 8:24 AM:

Just FYI, I am able to get things working with a that depends on {{google-cloud-datastore}}.

was (Author: mlambert):
Just FYI, I am able to get things working with a that depends on `google-cloud-datastore`.

> Failure to build --requirements.txt when it uses google protobuf
> ----------------------------------------------------------------
>                 Key: BEAM-1790
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py
>            Reporter: Mike Lambert
>            Assignee: Ahmet Altay
>              Labels: build, requirements
> I am running with {{--requirements_file requirements.txt}}, which contains:
> {noformat}
> google-cloud-datastore
> {noformat}
> Unfortunately, when attempting to run this on the cloud dataflow, I get the following
error trying to build the requirements:
> {noformat}
> Collecting setuptools (from protobuf>=3.0.0->google-cloud-core<0.24dev,>=0.23.1->google-cloud-datastore->-r
requirements.txt (line 3))
>   File was already downloaded /var/folders/94/wngs1jw91_n2_jjjrfljtqrc0000gn/T/dataflow-requirements-cache/
>     Complete output from command python egg_info:
>     Traceback (most recent call last):
>       File "<string>", line 1, in <module>
>       File "setuptools/", line 12, in <module>
>         import setuptools.version
>       File "setuptools/", line 1, in <module>
>         import pkg_resources
>       File "pkg_resources/", line 70, in <module>
>         import packaging.version
>     ImportError: No module named packaging.version
> {noformat}
> Looking online , it appears this is due
to "pip asking setuptools to build itself (from source dist), which is no longer supported."
> I'm not sure what the correct fix is here...since protobuf depends on setuptools, and
a lot of Google libraries depend on protobuf. Seems there is no way to whitelist protobuf/setuptools
as being "provided" by the beam runtime (ie
> I'm going to try using my own next and see if I can skirt around the issue,
but this definitely seems like a bug with beam's requirements packager asking for too much?
> In the case of GCE, I compile my dependencies into a docker image that extends the base
GCE images (and lets me use binary installs), not sure something like that would work here?

This message was sent by Atlassian JIRA

View raw message