arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xander Dunn" <xan...@xander.ai>
Subject Re: [Python] pyarrow.gandiva unavailable on Ubuntu?
Date Tue, 13 Apr 2021 22:05:51 GMT
One odd thing I noticed is that on Mac, calling ` conda install -c conda-forge pyarrow ` defaults
to installing pyarrow 3.0.0. However, on Ubuntu it defaults to installing pyarrow 0.11.1,
which is why above I've been specifying '==3.0.0` on all of the conda installs. I don't know
why conda on Ubuntu is attempting to install pyarrow 0.11.1 rather than latest.

On Tue, Apr 13, 2021 at 1:45 PM, Xander Dunn < xander@xander.ai > wrote:

> 
> ```
> 
> $ conda create -n fresh_env python=3.7
> 
> $ conda activate fresh_env
> 
> $ conda install -c conda-forge pyarrow==3.0.0
> 
> $ python
> 
> >>> import pyarrow as pa
> 
> >>> pa.__version__
> 
> '3.0.0'
> 
> >>> pa.__file__
> 
> '/ home/ xander/ anaconda3/ envs/ fresh_env/ lib/ python3. 7/ site-packages/
> pyarrow/ __init__. py (
> http://home/xander/anaconda3/envs/fresh_env/lib/python3.7/site-packages/pyarrow/__init__.py
> ) '
> 
> >>> import pyarrow.plasma
> 
> >>> import pyarrow.gandiva
> 
> Traceback (most recent call last):
> 
> File "<stdin>", line 1, in <module>
> 
> ModuleNotFoundError: No module named 'pyarrow.gandiva'
> 
> ```
> 
> Same result in a fresh conda environment. The library is also still
> missing from the installed location:
> 
> ```
> 
> $ l
> /home/xander/anaconda3/envs/fresh_env/lib/python3.7/site-packages/pyarrow/
> | grep gandiva
> 
> -rw-rw-r-- 3 xander xander  17K Jan 18 14:00 gandiva.pyx
> 
> ```
> 
> 
> 
> And a correction to what I wrote earlier, I'm on Ubuntu 18.04:
> 
> ```
> 
> $ lsb_release -a
> 
> No LSB modules are available.
> 
> Distributor ID: Ubuntu
> 
> Description:    Ubuntu 18.04.5 LTS
> 
> Release:        18.04
> 
> Codename:       bionic
> 
> $ conda --version
> 
> conda 4.10.0
> 
> ```
> 
> 
> 
> I do have several arrow packages installed via apt, but I don't think
> that's interfering:
> 
> ```
> 
> apache-arrow-archive-keyring/bionic,now 3.0.0-1 all [installed]
> 
> gir1.2-arrow-1.0/bionic,now 3.0.0-1 amd64 [installed,automatic]
> 
> libarrow-dev/bionic,now 3.0.0-1 amd64 [installed]
> 
> libarrow-glib-dev/bionic,now 3.0.0-1 amd64 [installed,automatic]
> 
> libarrow-glib300/bionic,now 3.0.0-1 amd64 [installed,automatic]
> 
> libarrow300/bionic,now 3.0.0-1 amd64 [installed]
> 
> ```
> 
> 
> 
> Thanks for the help,
> 
> Xander
> 
> 
> 
> 
> 
> On Tue, Apr 13, 2021 at 1:25 PM, Wes McKinney < wesmckinn@ gmail. com (
> wesmckinn@gmail.com ) > wrote:
> 
>> It looks to me like you have the wheel installed, not the conda package.
>> Can you reproduce this on Ubuntu from a fresh conda environment?
>> 
>> On Tue, Apr 13, 2021 at 3:16 PM Xander Dunn < xander@ xander. ai (
>> xander@xander.ai ) > wrote:
>> 
>> 
>>> Typo. The issue remains present. From my Ubuntu machine just now:
>>> 
>>> ```
>>> 
>>> $ python
>>> 
>>> >>> import pyarrow as pa
>>> 
>>> >>> print(pa.__file__)
>>> 
>>> / home/ xander/ anaconda3/ envs/ plutus_model/ lib/ python3. 7/ site-packages/
>>> pyarrow/ __init__. py (
>>> http://home/xander/anaconda3/envs/plutus_model/lib/python3.7/site-packages/pyarrow/__init__.py
>>> )
>>> 
>>> >>> import pyarrow.plasma
>>> 
>>> >>> import pyarrow.gandiva
>>> 
>>> Traceback (most recent call last):
>>> 
>>> File "<stdin>", line 1, in <module>
>>> 
>>> ModuleNotFoundError: No module named 'pyarrow.gandiva'
>>> 
>>> ```
>>> 
>>> 
>>> 
>>> The .py I'm executing on both machines is identical. Works on mac. Not
>>> found on Ubuntu.
>>> 
>>> 
>>> 
>>> 
>>> On Tue, Apr 13, 2021 at 1:01 PM, Micah Kornfield < emkornfield@ gmail. com
>>> ( emkornfield@gmail.com ) > wrote:
>>> 
>>>> Hi Xander,
>>>> Was there autocorrect on this e-mail?  the second example shows "gondiva"
>>>> not "gandiva"
>>>> 
>>>> 
>>>> -Micah
>>>> 
>>>> On Tue, Apr 13, 2021 at 12:59 PM Xander Dunn < xander@ xander. ai (
>>>> xander@xander.ai ) > wrote:
>>>> 
>>>> 
>>>>> On my local macOS 11.2.3:
>>>>> 
>>>>> ```
>>>>> 
>>>>> $ python --version
>>>>> 
>>>>> Python 3.7.10
>>>>> 
>>>>> $ pip --version
>>>>> 
>>>>> pip 21.0.1 from
>>>>> /usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pip (python
>>>>> 3.7)
>>>>> 
>>>>> $ pip list | grep pyarrow
>>>>> 
>>>>> pyarrow                3.0.0
>>>>> 
>>>>> $ which python
>>>>> 
>>>>> /usr/local/anaconda3/envs/my_model/bin/python
>>>>> 
>>>>> $ python
>>>>> 
>>>>> >>> import pyarrow as pa
>>>>> 
>>>>> >>> print(pa.__file__)
>>>>> 
>>>>> / usr/ local/ anaconda3/ envs/ my_model/ lib/ python3. 7/ site-packages/
pyarrow/
>>>>> __init__. py (
>>>>> http://usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/__init__.py
>>>>> )
>>>>> 
>>>>> >>> import pyarrow.plasma
>>>>> 
>>>>> >>> import pyarrow.gandiva as ga
>>>>> 
>>>>> >>> print(ga.__file__)
>>>>> 
>>>>> / usr/ local/ anaconda3/ envs/ my_model/ lib/ python3. 7/ site-packages/
pyarrow/
>>>>> gandiva. cpython-37m-darwin. so (
>>>>> http://usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/gandiva.cpython-37m-darwin.so
>>>>> )
>>>>> 
>>>>> ```
>>>>> 
>>>>> 
>>>>> 
>>>>> On my Ubuntu 14.04 instance:
>>>>> 
>>>>> ```
>>>>> 
>>>>> $ python --version
>>>>> 
>>>>> Python 3.7.10
>>>>> 
>>>>> $ pip --version
>>>>> 
>>>>> pip 21.0.1 from
>>>>> /home/xander/anaconda3/envs/my_model/lib/python3.7/site-packages/pip
>>>>> (python 3.7)
>>>>> 
>>>>> $ pip list | grep pyarrow
>>>>> 
>>>>> pyarrow                3.0.0
>>>>> 
>>>>> $ which python
>>>>> 
>>>>> /home/xander/anaconda3/envs/my_model/bin/python
>>>>> 
>>>>> $ python
>>>>> 
>>>>> >>> import pyarrow as pa
>>>>> 
>>>>> >>> print(pa.__file__)
>>>>> 
>>>>> / home/ xander/ anaconda3/ envs/ my_model/ lib/ python3. 7/ site-packages/
>>>>> pyarrow/ __init__. py (
>>>>> http://home/xander/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/__init__.py
>>>>> )
>>>>> 
>>>>> >>> import pyarrow.plasma
>>>>> 
>>>>> >>> import pyarrow.gondiva
>>>>> 
>>>>> Traceback (most recent call last):
>>>>> 
>>>>> File "<stdin>", line 1, in <module>
>>>>> 
>>>>> ModuleNotFoundError: No module named 'pyarrow.gondiva'
>>>>> 
>>>>> ```
>>>>> 
>>>>> You can see that pyarrow.gondiva is found on mac but not on Ubuntu. Same
>>>>> Python version. Same pyarrow version. I installed both of them with `conda
>>>>> install -c conda-forge pyarrow==3.0.0`.
>>>>> 
>>>>> 
>>>>> 
>>>>> On Mac, I see the expected Cython file and library:
>>>>> 
>>>>> ```
>>>>> 
>>>>> $ l
>>>>> /usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/
|
>>>>> grep gandiva
>>>>> 
>>>>> -rwxrwxr-x   2 xander  staff   221K Apr  1 12:44 gandiva. cpython-37m-darwin.
>>>>> so ( http://gandiva.cpython-37m-darwin.so/ )
>>>>> 
>>>>> -rw-rw-r--   2 xander  staff    17K Jan 18 14:00 gandiva.pyx
>>>>> 
>>>>> ```
>>>>> 
>>>>> 
>>>>> 
>>>>> On Ubuntu, I see only the Cython file:
>>>>> 
>>>>> ```
>>>>> 
>>>>> $ l ~/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/ |
grep
>>>>> gandiva
>>>>> 
>>>>> -rw-rw-r-- 1 xander xander  17K Apr 13 12:28 gandiva.pyx
>>>>> 
>>>>> ```
>>>>> 
>>>>> 
>>>>> 
>>>>> Is this expected? Should I be able to import pyarrow.gandiva on Ubuntu?
>>>>> Everything is run on Ubuntu so if I make use of pyarrow.gandiva I'll
need
>>>>> to figure out how to call it.
>>>>> 
>>>>> 
>>>>> 
>>>>> It's mentioned here that it was removed from Python wheels but should
>>>>> still be available in the conda install: https:/ / issues. apache. org/
jira/
>>>>> browse/ ARROW-10154 ( https://issues.apache.org/jira/browse/ARROW-10154
).
>>>>> I'm not finding it in my Ubuntu conda install.
>>>>> 
>>>>> 
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Xander
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 
>
Mime
View raw message