arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suvayu Ali <fatkasuvayu+li...@gmail.com>
Subject Installing PyArrow on Amazon Linux
Date Sun, 02 Jul 2017 12:01:44 GMT
Hi Arrow devs,

I'm not sure if this is the correct place to ask, if not, please point
me in the right direction.

I wanted to use the HDFS client with PySpark (for now).  My Spark
cluster is on Amazon EMR, so the nodes use Amazon Linux (2017.03).  On
my dev machine (Fedora 25) with Python 3.5.1, a simple `pip3 install
pyarrow` works just fine, and I can use the HDFS client.  But when I do
the same on Amazon Linux, it complains about incompatible Python
version, then pip tries to download an old tarball (0.3), and finally
cmake fails with missing cmake files (PythonLibsNewConfig.cmake).

Here's a snippet from the failure with verbose on:

  $ sudo python3.5 -m pip -vvv install pyarrow
  [..]
    Skipping https://pypi.python.org/../pyarrow-0.4.1-cp35-cp35m-manylinux1_x86_64.whl#md5=..

    (from https://pypi.python.org/simple/pyarrow/) because it is not compatible with this
Python
    Skipping https://pypi.python.org/../pyarrow-0.4.0-cp27-cp27mu-manylinux1_x86_64.whl#md5=..
    (from https://pypi.python.org/simple/pyarrow/) because it is not compatible with this
Python
    Skipping https://pypi.python.org/../pyarrow-0.2.0-cp34-cp34m-manylinux1_x86_64.whl#md5=..
    (from https://pypi.python.org/simple/pyarrow/) because it is not compatible with this
Python
    Using version 0.3.0 (newest of versions: 0.3.0, 0.2.0)
  [..]

When I explicitly specify which wheel file to use, it fails like this:

  $ sudo python3.5 -m pip -vvv install https://pypi.python.org/../pyarrow-0.4.1-cp35-cp35m-manylinux1_x86_64.whl#md5=..
  pyarrow-0.4.1-cp35-cp35m-manylinux1_x86_64.whl is not a supported wheel on this platform.
  Exception information:
  Traceback (most recent call last):
    File "/usr/lib/python3.5/dist-packages/pip/basecommand.py", line 246, in main
      status = self.run(options, args)
    File "/usr/lib/python3.5/dist-packages/pip/commands/install.py", line 308, in run
      name, None, isolated=options.isolated_mode,
    File "/usr/lib/python3.5/dist-packages/pip/req/req_install.py", line 207, in from_line
      wheel.filename
  pip.exceptions.UnsupportedWheel: pyarrow-0.4.1-cp35-cp35m-manylinux1_x86_64.whl is not a
supported wheel on this platform.

Any ideas what is going wrong?  Thanks for any help.

Cheers,

-- 
Suvayu

Open source is the future. It sets us free.

Mime
View raw message