beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ahmet Altay (JIRA)" <>
Subject [jira] [Commented] (BEAM-1790) Failure to build --requirements.txt when it uses google protobuf
Date Thu, 23 Mar 2017 18:19:41 GMT


Ahmet Altay commented on BEAM-1790:

[~mlambert] Where do you get this error, is this a at job creation time or execution time?

I managed to reproduce this error with and old (unsupported by Beam) verison of pip (1.5.4).
Could you check your installed pip version?

> Failure to build --requirements.txt when it uses google protobuf
> ----------------------------------------------------------------
>                 Key: BEAM-1790
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py
>            Reporter: Mike Lambert
>            Assignee: Ahmet Altay
>              Labels: build, requirements
> I am running with {{--requirements_file requirements.txt}}, which contains:
> {noformat}
> google-cloud-datastore
> {noformat}
> Unfortunately, when attempting to run this on the cloud dataflow, I get the following
error trying to build the requirements:
> {noformat}
> Collecting setuptools (from protobuf>=3.0.0->google-cloud-core<0.24dev,>=0.23.1->google-cloud-datastore->-r
requirements.txt (line 3))
>   File was already downloaded /var/folders/94/wngs1jw91_n2_jjjrfljtqrc0000gn/T/dataflow-requirements-cache/
>     Complete output from command python egg_info:
>     Traceback (most recent call last):
>       File "<string>", line 1, in <module>
>       File "setuptools/", line 12, in <module>
>         import setuptools.version
>       File "setuptools/", line 1, in <module>
>         import pkg_resources
>       File "pkg_resources/", line 70, in <module>
>         import packaging.version
>     ImportError: No module named packaging.version
> {noformat}
> Looking online , it appears this is due
to "pip asking setuptools to build itself (from source dist), which is no longer supported."
> I'm not sure what the correct fix is here...since protobuf depends on setuptools, and
a lot of Google libraries depend on protobuf. Seems there is no way to whitelist protobuf/setuptools
as being "provided" by the beam runtime (ie
> I'm going to try using my own next and see if I can skirt around the issue,
but this definitely seems like a bug with beam's requirements packager asking for too much?
> In the case of GCE, I compile my dependencies into a docker image that extends the base
GCE images (and lets me use binary installs), not sure something like that would work here?

This message was sent by Atlassian JIRA

View raw message