beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ryan Williams (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (BEAM-4749) fastavro breaks macos tests
Date Sun, 15 Jul 2018 20:37:00 GMT

    [ https://issues.apache.org/jira/browse/BEAM-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16544664#comment-16544664
] 

Ryan Williams edited comment on BEAM-4749 at 7/15/18 8:36 PM:
--------------------------------------------------------------

{quote}I do not know if I ran the xcode command before, I can try to run again with that.
{quote}
OK, lmk how that goes if you try it.

Thinking on it more, {{brew install gcc}} is probably better, if we're just talking about
how to get a working {{gcc}} on OSX.

macOS's system "{{gcc}}" is actually {{clang}}, and requires {{xcode-select --install}} for
some things (which is more heavyweight, e.g. requires root).

Either way, it seems reasonable to me to require that users have a working {{gcc}} before
building Beam; we already expect them to bring their own {{git}}, {{java}}, {{python}}, etc.,
right?

lmk if that's not reasonable though, or I'm misunderstanding the environment you / others
were hitting this issue in.
{quote}Is it possible to gracefully fallback to avro in client environments when fastavro
is not installed.
{quote}
You mean if users set {{use_fastavro=True}}, but don't have {{fastavro}} installed? (all fastavro
code-paths are off by default / controlled by CLI flags today)

As discussed above, I think having a simple story for users to get {{fastavro}} installed
will be straightforward, and that'd be a better solution, but yea, we can explore falling-back
to apache/avro when {{use_fastavro=True}} otherwise, if you think that's best.

Thanks again for digging in to it!


was (Author: rdub):
{quote}I do not know if I ran the xcode command before, I can try to run again with that.
{quote}
OK, lmk how that goes if you try it.

Thinking on it more, {{brew install gcc}} is probably better, if we're just talking about
how to get a working {{gcc}} on OSX.

macOS's system "{{gcc}}" is actually {{clang}}, and requires {{xcode-select --install}} for
some things (which is more heavyweight, e.g. requires root).

Either way, it seems reasonable to me to require that users have a working {{gcc}} before
building Beam; we already expect them to bring their own {{git}}, {{java}}, {{python}}, etc.,
right?

lmk if that's not reasonable though, or I'm misunderstanding the environment you / others
were hitting this issue in.

 

 
{quote}Is it possible to gracefully fallback to avro in client environments when fastavro
is not installed.
{quote}
You mean if users set {{use_fastavro=True}}, but don't have {{fastavro}} installed? (all fastavro
code-paths are off by default / controlled by CLI flags today)

As discussed above, I think having a simple story for users to get {{fastavro}} installed
will be straightforward, and that'd be a better solution, but yea, we can explore falling-back
to apache/avro when {{use_fastavro=True}} otherwise, if you think that's best.

Thanks again for digging in to it!

> fastavro breaks macos tests
> ---------------------------
>
>                 Key: BEAM-4749
>                 URL: https://issues.apache.org/jira/browse/BEAM-4749
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-py-core
>            Reporter: Ahmet Altay
>            Assignee: Ryan Williams
>            Priority: Major
>          Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Recent addition of the fastavro dependency is breaking python linter in macos. At least
for some cases, because it requires a compiler.
> Could we optionally depend on fastavro, and fallback to regular avro package?
>  
> Log:
> *> Task :beam-sdks-python:lintPy27*
> ERROR: invocation failed (exit code 1), logfile: /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/log/py27-lint-2.log
> ERROR: actionid: py27-lint
> msg: installpkg
> cmdargs: ['/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python',
'/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/pip', 'install', '/Users/relax/beam-gradle/beam/sdks/python/target/.tox/dist/apache-beam-2.6.0.dev0.zip[test]']
>  
> Processing ./target/.tox/dist/apache-beam-2.6.0.dev0.zip
> Collecting avro<2.0.0,>=1.8.1 (from apache-beam==2.6.0.dev0)
> Collecting crcmod<2.0,>=1.7 (from apache-beam==2.6.0.dev0)
> Collecting dill==0.2.6 (from apache-beam==2.6.0.dev0)
> Collecting fastavro==0.19.7 (from apache-beam==2.6.0.dev0)
>   Using cached [https://files.pythonhosted.org/packages/a7/0a/b08ba5cef63c675e8442c2bf1cbcef90c8b9f824be2202d492f0cedb0913/fastavro-0.19.7.tar.gz]
> Collecting grpcio<2,>=1.8 (from apache-beam==2.6.0.dev0)
>   Using cached [https://files.pythonhosted.org/packages/66/89/4a90caabd51c17686cbb48a9bbe8c592c4be929c0d2542d2ffde76b0d671/grpcio-1.13.0-cp27-cp27m-macosx_10_12_x86_64.whl]
> Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam==2.6.0.dev0)
> Collecting httplib2<=0.11.3,>=0.8 (from apache-beam==2.6.0.dev0)
> Collecting mock<3.0.0,>=1.0.1 (from apache-beam==2.6.0.dev0)
>   Using cached [https://files.pythonhosted.org/packages/e6/35/f187bdf23be87092bd0f1200d43d23076cee4d0dec109f195173fd3ebc79/mock-2.0.0-py2.py3-none-any.whl]
> Collecting oauth2client<5,>=2.0.1 (from apache-beam==2.6.0.dev0)
>   Using cached [https://files.pythonhosted.org/packages/82/d8/3eab58811282ac7271a081ba5c0d4b875ce786ca68ce43e2a62ade32e9a8/oauth2client-4.1.2-py2.py3-none-any.whl]
> Collecting protobuf<4,>=3.5.0.post1 (from apache-beam==2.6.0.dev0)
>   Using cached [https://files.pythonhosted.org/packages/4f/56/a21f2d077ceae7fd521c0ed31fb8bb1c7f13ffbb09bf7dd27de6cf6bad08/protobuf-3.6.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl]
> Collecting pytz<=2018.4,>=2018.3 (from apache-beam==2.6.0.dev0)
>   Using cached [https://files.pythonhosted.org/packages/dc/83/15f7833b70d3e067ca91467ca245bae0f6fe56ddc7451aa0dc5606b120f2/pytz-2018.4-py2.py3-none-any.whl]
> Collecting pyyaml<4.0.0,>=3.12 (from apache-beam==2.6.0.dev0)
>   Using cached [https://files.pythonhosted.org/packages/4a/85/db5a2df477072b2902b0eb892feb37d88ac635d36245a72a6a69b23b383a/PyYAML-3.12.tar.gz]
> Collecting pyvcf<0.7.0,>=0.6.8 (from apache-beam==2.6.0.dev0)
> Requirement already satisfied: six<1.12,>=1.9 in ./target/.tox/py27-lint/lib/python2.7/site-packages
(from apache-beam==2.6.0.dev0) (1.11.0)
> Collecting typing<3.7.0,>=3.6.0 (from apache-beam==2.6.0.dev0)
>   Using cached [https://files.pythonhosted.org/packages/0d/4d/4e5985d075d241d686a1663fa1f88b61d544658d08c1375c7c6aac32afc3/typing-3.6.4-py2-none-any.whl]
> Collecting futures<4.0.0,>=3.1.1 (from apache-beam==2.6.0.dev0)
>   Using cached [https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl]
> Requirement already satisfied: future<1.0.0,>=0.16.0 in ./target/.tox/py27-lint/lib/python2.7/site-packages
(from apache-beam==2.6.0.dev0) (0.16.0)
> Collecting nose>=1.3.7 (from apache-beam==2.6.0.dev0)
>   Using cached [https://files.pythonhosted.org/packages/99/4f/13fb671119e65c4dce97c60e67d3fd9e6f7f809f2b307e2611f4701205cb/nose-1.3.7-py2-none-any.whl]
> Collecting pyhamcrest<2.0,>=1.9 (from apache-beam==2.6.0.dev0)
>   Using cached [https://files.pythonhosted.org/packages/9a/d5/d37fd731b7d0e91afcc84577edeccf4638b4f9b82f5ffe2f8b62e2ddc609/PyHamcrest-1.9.0-py2.py3-none-any.whl]
> Requirement already satisfied: setuptools>=18.0 in ./target/.tox/py27-lint/lib/python2.7/site-packages
(from fastavro==0.19.7->apache-beam==2.6.0.dev0) (39.2.0)
> Requirement already satisfied: enum34>=1.0.4 in ./target/.tox/py27-lint/lib/python2.7/site-packages
(from grpcio<2,>=1.8->apache-beam==2.6.0.dev0) (1.1.6)
> Collecting docopt (from hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
> Collecting requests>=2.7.0 (from hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
>   Using cached [https://files.pythonhosted.org/packages/65/47/7e02164a2a3db50ed6d8a6ab1d6d60b69c4c3fdf57a284257925dfc12bda/requests-2.19.1-py2.py3-none-any.whl]
> Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam==2.6.0.dev0)
>   Using cached [https://files.pythonhosted.org/packages/b3/5d/c196041ffdf3e34ba206db6d61d1f893a75e1f3435699ade9bd65e089a3d/pbr-4.0.4-py2.py3-none-any.whl]
> Collecting funcsigs>=1; python_version < "3.3" (from mock<3.0.0,>=1.0.1->apache-beam==2.6.0.dev0)
>   Using cached [https://files.pythonhosted.org/packages/69/cb/f5be453359271714c01b9bd06126eaf2e368f1fddfff30818754b5ac2328/funcsigs-1.0.2-py2.py3-none-any.whl]
> Collecting rsa>=3.1.4 (from oauth2client<5,>=2.0.1->apache-beam==2.6.0.dev0)
>   Using cached [https://files.pythonhosted.org/packages/e1/ae/baedc9cb175552e95f3395c43055a6a5e125ae4d48a1d7a924baca83e92e/rsa-3.4.2-py2.py3-none-any.whl]
> Collecting pyasn1>=0.1.7 (from oauth2client<5,>=2.0.1->apache-beam==2.6.0.dev0)
>   Using cached [https://files.pythonhosted.org/packages/a0/70/2c27740f08e477499ce19eefe05dbcae6f19fdc49e9e82ce4768be0643b9/pyasn1-0.4.3-py2.py3-none-any.whl]
> Collecting pyasn1-modules>=0.0.5 (from oauth2client<5,>=2.0.1->apache-beam==2.6.0.dev0)
>   Using cached [https://files.pythonhosted.org/packages/19/02/fa63f7ba30a0d7b925ca29d034510fc1ffde53264b71b4155022ddf3ab5d/pyasn1_modules-0.2.2-py2.py3-none-any.whl]
> Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
>   Using cached [https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl]
> Collecting idna<2.8,>=2.5 (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
>   Using cached [https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl]
> Collecting urllib3<1.24,>=1.21.1 (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
>   Using cached [https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl]
> Collecting certifi>=2017.4.17 (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
>   Using cached [https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl]
> Building wheels for collected packages: apache-beam, fastavro, pyyaml
>   Running setup.py bdist_wheel for apache-beam: started
>   Running setup.py bdist_wheel for apache-beam: finished with status 'done'
>   Stored in directory: /Users/relax/Library/Caches/pip/wheels/a3/18/08/60ce57e27e8b04460419c9463b9b6da19a8aab9fb64afc612f
>   Running setup.py bdist_wheel for fastavro: started
>   Running setup.py bdist_wheel for fastavro: finished with status 'error'
>   Complete output from command /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python
-u -c "import setuptools, tokenize;__file__='/private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-install-WX5Ntf/fastavro/setup.py';f=getattr(tokenize,
'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__,
'exec'))" bdist_wheel -d /private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-wheel-p6UGao
--python-tag cp27:
>   running bdist_wheel
>   running build
>   running build_py
>   creating build
>   creating build/lib.macosx-10.4-x86_64-2.7
>   creating build/lib.macosx-10.4-x86_64-2.7/fastavro
>   copying fastavro/_schema_common.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>   copying fastavro/_schema_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>   copying fastavro/write.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>   copying fastavro/_write_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>   copying fastavro/__init__.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>   copying fastavro/_read_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>   copying fastavro/_read_common.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>   copying fastavro/_validate_common.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>   copying fastavro/_validation_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>   copying fastavro/six.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>   copying fastavro/_timezone.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>   copying fastavro/__main__.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>   copying fastavro/const.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>   copying fastavro/schema.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>   copying fastavro/read.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>   copying fastavro/validation.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>   running build_ext
>   building 'fastavro._read' extension
>   creating build/temp.macosx-10.4-x86_64-2.7
>   creating build/temp.macosx-10.4-x86_64-2.7/fastavro
>   gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes
-I/Library/GoogleCorpSupport/Python/2.7/include/python2.7 -c fastavro/_read.c -o build/temp.macosx-10.4-x86_64-2.7/fastavro/_read.o
>   xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools),
missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
>   error: command 'gcc' failed with exit status 1
>   
>   ----------------------------------------
>   Failed building wheel for fastavro
>   Running setup.py clean for fastavro
>   Running setup.py bdist_wheel for pyyaml: started
>   Running setup.py bdist_wheel for pyyaml: finished with status 'done'
>   Stored in directory: /Users/relax/Library/Caches/pip/wheels/03/05/65/bdc14f2c6e09e82ae3e0f13d021e1b6b2481437ea2f207df3f
> Successfully built apache-beam pyyaml
> Failed to build fastavro
> Installing collected packages: avro, crcmod, dill, fastavro, futures, grpcio, docopt,
chardet, idna, urllib3, certifi, requests, hdfs, httplib2, pbr, funcsigs, mock, pyasn1, rsa,
pyasn1-modules, oauth2client, protobuf, pytz, pyyaml, pyvcf, typing, nose, pyhamcrest, apache-beam
>   Running setup.py install for fastavro: started
>     Running setup.py install for fastavro: finished with status 'error'
>     Complete output from command /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python
-u -c "import setuptools, tokenize;__file__='/private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-install-WX5Ntf/fastavro/setup.py';f=getattr(tokenize,
'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__,
'exec'))" install --record /private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-record-sAEfXi/install-record.txt
--single-version-externally-managed --compile --install-headers /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/include/site/python2.7/fastavro:
>     running install
>     running build
>     running build_py
>     creating build
>     creating build/lib.macosx-10.4-x86_64-2.7
>     creating build/lib.macosx-10.4-x86_64-2.7/fastavro
>     copying fastavro/_schema_common.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>     copying fastavro/_schema_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>     copying fastavro/write.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>     copying fastavro/_write_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>     copying fastavro/__init__.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>     copying fastavro/_read_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>     copying fastavro/_read_common.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>     copying fastavro/_validate_common.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>     copying fastavro/_validation_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>     copying fastavro/six.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>     copying fastavro/_timezone.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>     copying fastavro/__main__.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>     copying fastavro/const.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>     copying fastavro/schema.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>     copying fastavro/read.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>     copying fastavro/validation.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
>     running build_ext
>     building 'fastavro._read' extension
>     creating build/temp.macosx-10.4-x86_64-2.7
>     creating build/temp.macosx-10.4-x86_64-2.7/fastavro
>     gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes
-I/Library/GoogleCorpSupport/Python/2.7/include/python2.7 -c fastavro/_read.c -o build/temp.macosx-10.4-x86_64-2.7/fastavro/_read.o
>     xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools),
missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
>     error: command 'gcc' failed with exit status 1
>     
>     ----------------------------------------
> Command "/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python -u
-c "import setuptools, tokenize;__file__='/private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-install-WX5Ntf/fastavro/setup.py';f=getattr(tokenize,
'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__,
'exec'))" install --record /private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-record-sAEfXi/install-record.txt
--single-version-externally-managed --compile --install-headers /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/include/site/python2.7/fastavro"
failed with error code 1 in /private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-install-WX5Ntf/fastavro/
>  
> py27-lint installed: astroid==1.6.5,avro==1.8.2,backports.functools-lru-cache==1.5,configparser==3.5.0,crcmod==1.7,dill==0.2.6,enum34==1.1.6,flake8==3.5.0,future==0.16.0,isort==4.2.15,lazy-object-proxy==1.3.1,mccabe==0.6.1,pycodestyle==2.3.1,pyflakes==1.6.0,pylint==1.7.2,singledispatch==3.4.0.3,six==1.11.0,wrapt==1.10.11
> ___________________________________ summary ____________________________________
> ERROR:   py27-lint: InvocationError for command /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python
/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/pip install /Users/relax/beam-gradle/beam/sdks/python/target/.tox/dist/apache-beam-2.6.0.dev0.zip[test]
(see /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/log/py27-lint-2.log)
(exited with code 1)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message