beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mike Lambert (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (BEAM-1790) Failure to build --requirements.txt when it uses google protobuf
Date Thu, 23 Mar 2017 08:25:41 GMT

    [ https://issues.apache.org/jira/browse/BEAM-1790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15937917#comment-15937917
] 

Mike Lambert edited comment on BEAM-1790 at 3/23/17 8:24 AM:
-------------------------------------------------------------

Just FYI, I am able to get things working with a setup.py that depends on {{google-cloud-datastore}}.


was (Author: mlambert):
Just FYI, I am able to get things working with a setup.py that depends on `google-cloud-datastore`.

> Failure to build --requirements.txt when it uses google protobuf
> ----------------------------------------------------------------
>
>                 Key: BEAM-1790
>                 URL: https://issues.apache.org/jira/browse/BEAM-1790
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py
>            Reporter: Mike Lambert
>            Assignee: Ahmet Altay
>              Labels: build, requirements
>
> I am running with {{--requirements_file requirements.txt}}, which contains:
> {noformat}
> google-cloud-datastore
> {noformat}
> Unfortunately, when attempting to run this on the cloud dataflow, I get the following
error trying to build the requirements:
> {noformat}
> Collecting setuptools (from protobuf>=3.0.0->google-cloud-core<0.24dev,>=0.23.1->google-cloud-datastore->-r
requirements.txt (line 3))
>   File was already downloaded /var/folders/94/wngs1jw91_n2_jjjrfljtqrc0000gn/T/dataflow-requirements-cache/setuptools-34.3.2.zip
>     Complete output from command python setup.py egg_info:
>     Traceback (most recent call last):
>       File "<string>", line 1, in <module>
>       File "setuptools/__init__.py", line 12, in <module>
>         import setuptools.version
>       File "setuptools/version.py", line 1, in <module>
>         import pkg_resources
>       File "pkg_resources/__init__.py", line 70, in <module>
>         import packaging.version
>     ImportError: No module named packaging.version
> {noformat}
> Looking online https://github.com/pypa/setuptools/issues/937 , it appears this is due
to "pip asking setuptools to build itself (from source dist), which is no longer supported."
> I'm not sure what the correct fix is here...since protobuf depends on setuptools, and
a lot of Google libraries depend on protobuf. Seems there is no way to whitelist protobuf/setuptools
as being "provided" by the beam runtime (ie https://github.com/pypa/pip/issues/3090).
> I'm going to try using my own setup.py next and see if I can skirt around the issue,
but this definitely seems like a bug with beam's requirements packager asking for too much?
> In the case of GCE, I compile my dependencies into a docker image that extends the base
GCE images (and lets me use binary installs), not sure something like that would work here?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message