arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vibhatha Abeykoon <vibha...@gmail.com>
Subject Re: PyArrow building from source issue
Date Sat, 02 May 2020 23:01:14 GMT
One thing I noticed,

-DCMAKE_INSTALL_LIBDIR=lib \

this path is relative to ARROW_HOME, correct me if I am wrong?

So with this source build, it doesn't necessarily copy files to /usr/lib ?

With Regards,
Vibhatha Abeykoon


On Sat, May 2, 2020 at 6:12 PM Vibhatha Abeykoon <vibhatha@gmail.com> wrote:

> So far this is the updated steps,
>
> pwd => /home/vibhatha/sandbox/arrow/repos
>
> export PYTHON_EXEC=/home/vibhatha/sandbox/arrow/repos/ENVARROW/bin/python3
> export LIB_DIR=$(pwd)/libs
>
> mkdir $LIB_DIR
>
> ## ENV CONFIGS
>
> export ARROW_HOME=export ARROW_HOME=$(pwd)/dist
> export LD_LIBRARY_PATH=$(pwd)/dist/lib:$LD_LIBRARY_PATH
>
> ## MAKE BUILD DIRS
>
> mkdir arrow/cpp/build
> pushd arrow/cpp/build
>
> cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
>       -DCMAKE_INSTALL_LIBDIR=$LIB_DIR \
>       -DARROW_WITH_BZ2=OFF \
>       -DARROW_WITH_ZLIB=OFF \
>       -DARROW_WITH_ZSTD=OFF \
>       -DARROW_WITH_LZ4=OFF \
>       -DARROW_WITH_SNAPPY=OFF \
>       -DARROW_WITH_BROTLI=OFF \
>       -DARROW_PARQUET=OFF \
>       -DARROW_PYTHON=ON \
>       -DARROW_BUILD_TESTS=ON \
>       -DPYTHON_EXECUTABLE=$PYTHON_EXEC \
>       ..
>
> make -j4
> make install
>
> popd
>
> export LD_LIBRARY_PATH=$LIB_DIR:$LD_LIBRARY_PATH
>
> pushd arrow/python
> PYARROW_CMAKE_OPTIONS="-DCMAKE_MODULE_PATH="${LIB_DIR}"/cmake/arrow"
> python3 setup.py install
>
>
> *Getting an error *
>
> CMake Error at
> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146
> (message):
>   Could NOT find Arrow (missing: ARROW_LIB_DIR) (found version "0.16.0")
> Call Stack (most recent call first):
>
> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393
> (_FPHSA_FAILURE_MESSAGE)
>   /home/vibhatha/sandbox/arrow/repos/libs/cmake/arrow/FindArrow.cmake:420
> (find_package_handle_standard_args)
>
> /home/vibhatha/sandbox/arrow/repos/libs/cmake/arrow/FindArrowPython.cmake:46
> (find_package)
>   CMakeLists.txt:204 (find_package)
>
>
> Then, I added the following in FindArrow.cmake
>
> /home/vibhatha/sandbox/arrow/repos/libs/cmake/arrow/FindArrow.cmake
>
> set(ARROW_LIB_DIR /home/vibhatha/sandbox/arrow/repos/libs)
>
> Re-ran the script,
>
> *Getting Error*
>
> CMake Error at
> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146
> (message):
>   Could NOT find ArrowPython (missing: ARROW_PYTHON_LIB_DIR) (found version
>   "0.16.0")
> Call Stack (most recent call first):
>
> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393
> (_FPHSA_FAILURE_MESSAGE)
>
> /home/vibhatha/sandbox/arrow/repos/libs/cmake/arrow/FindArrowPython.cmake:78
> (find_package_handle_standard_args)
>   CMakeLists.txt:204 (find_package)
>
> Added the following
> /home/vibhatha/sandbox/arrow/repos/libs/cmake/arrow/FindArrowPython.cmake
>
> set(ARROW_PYTHON_LIB_DIR /home/vibhatha/sandbox/arrow/repos/libs)
>
>
> *It seems a path I added could be wrong or a build flag is missing*
>
> -- Arrow version: 0.16.0 (HOME: /home/vibhatha/sandbox/arrow/repos/dist)
> -- Arrow SO and ABI version: 16
> -- Arrow full SO version: 16.0.0
>
>
>
>
>
>
> *-- Found the Arrow core shared library: ARROW_shared_lib-NOTFOUND-- Found
> the Arrow core import library: -- Found the Arrow core static library:
> ARROW_static_lib-NOTFOUND-- Found the Arrow Python by HOME:
> /home/vibhatha/sandbox/arrow/repos/dist-- Found the Arrow Python shared
> library: ARROW_PYTHON_shared_lib-NOTFOUND-- Found the Arrow Python import
> library: -- Found the Arrow Python static library:
> ARROW_PYTHON_static_lib-NOTFOUND*
> -- Configuring done
> -- Generating done
> -- Build files have been written to:
> /home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8
> -- Finished cmake for pyarrow
> -- Running cmake --build for pyarrow
> cmake --build . --config release --
> [  6%] Compiling Cython CXX source for _compute...
> [  6%] Built target _compute_pyx
> Scanning dependencies of target _compute
> [ 13%] Building CXX object CMakeFiles/_compute.dir/_compute.cpp.o
> [ 20%] Linking CXX shared module release/_
> compute.cpython-38-x86_64-linux-gnu.so
>
>
> */usr/bin/ld: cannot find -larrow_shared/usr/bin/ld: cannot find
> -larrow_python_shared*collect2: error: ld returned 1 exit status
> make[2]: *** [CMakeFiles/_compute.dir/build.make:84: release/_
> compute.cpython-38-x86_64-linux-gnu.so] Error 1
> make[1]: *** [CMakeFiles/Makefile2:89: CMakeFiles/_compute.dir/all] Error 2
> make: *** [Makefile:84: all] Error 2
> error: command 'cmake' failed with exit status 2
>
> The log also shows that the shared or static libraries have not been
> found,
>
>
> With Regards,
> Vibhatha Abeykoon
>
>
> On Fri, May 1, 2020 at 8:23 PM Vibhatha Abeykoon <vibhatha@gmail.com>
> wrote:
>
>> Thank you for the clarification.
>>
>> On Fri, May 1, 2020 at 8:10 PM Sutou Kouhei <kou@clear-code.com> wrote:
>>
>>> Hi,
>>>
>>> It'll work.
>>> Note that the LD_LIBRARY_PATH is the environment variable at
>>> run time. You need to specify the correct ARROW_HOME or
>>> -DCMAKE_MODULE_PATH at build time too.
>>>
>>>
>>> Thanks,
>>> --
>>> kou
>>>
>>> In <CABC-QT=U_+AE8LaMmHwfhJP+4wvVKJ0Y-z7styK0TxtrJyZdNQ@mail.gmail.com>
>>>   "Re: PyArrow building from source issue" on Fri, 1 May 2020 18:13:54
>>> -0400,
>>>   Vibhatha Abeykoon <vibhatha@gmail.com> wrote:
>>>
>>> > I will elaborate a couple of reasons,
>>> >
>>> > When there are a couple of versions of Arrow, used for different
>>> projects
>>> > depending on various development choices, it is convenient for me to
>>> keep
>>> > them pointed towards a folder of my choice.
>>> > Then refer to it and continue the work. Correct me if I am wrong, what
>>> if I
>>> > point to this folder of my choice and add it to the LD_LIBRARY_PATH.
>>> > Will this cause issues?
>>> >
>>> > With Regards,
>>> > Vibhatha Abeykoon
>>> >
>>> >
>>> > On Fri, May 1, 2020 at 6:09 PM Vibhatha Abeykoon <vibhatha@gmail.com>
>>> wrote:
>>> >
>>> >> Okay, thank you for your response.
>>> >>
>>> >> With Regards,
>>> >> Vibhatha Abeykoon
>>> >>
>>> >>
>>> >> On Fri, May 1, 2020 at 5:17 PM Sutou Kouhei <kou@clear-code.com>
>>> wrote:
>>> >>
>>> >>> Hi,
>>> >>>
>>> >>> > I used these settings, I still want the libs to be in a custom
>>> >>> directory,
>>> >>> > not in lib/
>>> >>>
>>> >>> Why? We don't recommend it if you don't a CMake expert.
>>> >>>
>>> >>> You can't do this if you use ARROW_HOME environment
>>> >>> variable.
>>> >>>
>>> >>> You may able to do this by removing ARROW_HOME environment
>>> >>> variable and adding
>>> >>>
>>> >>>
>>> PYARROW_CMAKE_OPTIONS="-DCMAKE_MODULE_PATH=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist/lib/cmake/arrow"
>>> >>> environment variable or something.
>>> >>>
>>> >>>
>>> >>> Thanks,
>>> >>> --
>>> >>> kou
>>> >>>
>>> >>> In <CABC-QTmsekZwjWG4X=
>>> aWrfQZgYi+qLdyEc_kvbN2njgtwJLofw@mail.gmail.com>
>>> >>>   "Re: PyArrow building from source issue" on Fri, 1 May 2020
>>> 15:05:51
>>> >>> -0400,
>>> >>>   Vibhatha Abeykoon <vibhatha@gmail.com> wrote:
>>> >>>
>>> >>> > export ARROW_HOME=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist
>>> >>> > export LD_LIBRARY_PATH=$ARROW_HOME/lib:$LD_LIBRARY_PATH
>>> >>> >
>>> >>> > export PYARROW_WITH_PARQUET=1
>>> >>> > export PYARROW_WITH_PYTHON=1
>>> >>> > export PYARROW_WITH_BUILD_TESTS=1
>>> >>> >
>>> >>> > cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
>>> >>> >
>>> >>> >
>>> >>>
>>> -DCMAKE_INSTALL_LIBDIR=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs
>>> >>> > \
>>> >>> >       -DARROW_WITH_BZ2=OFF \
>>> >>> >       -DARROW_WITH_ZLIB=OFF \
>>> >>> >       -DARROW_WITH_ZSTD=OFF \
>>> >>> >       -DARROW_WITH_LZ4=OFF \
>>> >>> >       -DARROW_WITH_SNAPPY=OFF \
>>> >>> >       -DARROW_WITH_BROTLI=OFF \
>>> >>> >       -DARROW_PARQUET=ON \
>>> >>> >       -DARROW_PYTHON=ON \
>>> >>> >       -DARROW_BUILD_TESTS=ON \
>>> >>> >
>>> >>> >
>>> >>>
>>> -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3
>>> >>> > \
>>> >>> >       ..
>>> >>> >
>>> >>> >
>>> >>> > I used these settings, I still want the libs to be in a custom
>>> >>> directory,
>>> >>> > not in lib/
>>> >>> >
>>> >>> > Does it make things not work?
>>> >>> >
>>> >>> > Now I get the following error,
>>> >>> >
>>> >>> > python setup.py build_ext --inplace
>>> >>> > WARNING: The wheel package is not available.
>>> >>> > running build_ext
>>> >>> > -- Running cmake for pyarrow
>>> >>> > cmake
>>> >>> -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/ENVARROW/bin/python
>>> >>> >  -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=off
>>> >>> > -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off
>>> >>> > -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=on
>>> >>> > -DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off
>>> >>> -DPYARROW_BUILD_HDFS=off
>>> >>> > -DPYARROW_USE_TENSORFLOW=off -DPYARROW_BUNDLE_ARROW_CPP=off
>>> >>> > -DPYARROW_BUNDLE_BOOST=off -DPYARROW_GENERATE_COVERAGE=off
>>> >>> > -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_PARQUET_USE_SHARED=on
>>> >>> > -DCMAKE_BUILD_TYPE=release
>>> >>> /home/vibhatha/sandbox/arrow/repos/arrow/python
>>> >>> > -- System processor: x86_64
>>> >>> > -- Arrow build warning level: PRODUCTION
>>> >>> > Using ld linker
>>> >>> > Configured for RELEASE build (set with cmake
>>> >>> > -DCMAKE_BUILD_TYPE={release,debug,...})
>>> >>> > -- Build Type: RELEASE
>>> >>> > -- Build output directory:
>>> >>> >
>>> >>>
>>> /home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/release
>>> >>> > -- Arrow version: 0.18.0 (HOME:
>>> >>> > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist)
>>> >>> > -- Arrow SO and ABI version: 18
>>> >>> > -- Arrow full SO version: 18.0.0
>>> >>> > -- Found the Arrow core shared library:
>>> >>> >
>>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so
>>> >>> > -- Found the Arrow core import library:
>>> >>> >
>>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so
>>> >>> > -- Found the Arrow core static library:
>>> >>> > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.a
>>> >>> > -- Found the Arrow Python by HOME:
>>> >>> > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist
>>> >>> > -- Found the Arrow Python shared library:
>>> >>> >
>>> >>>
>>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow_python.so
>>> >>> > -- Found the Arrow Python import library:
>>> >>> >
>>> >>>
>>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow_python.so
>>> >>> > -- Found the Arrow Python static library:
>>> >>> >
>>> >>>
>>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow_python.a
>>> >>> > CMake Error at
>>> >>> >
>>> >>>
>>> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146
>>> >>> > (message):
>>> >>> >   Could NOT find Parquet (missing: PARQUET_LIB_DIR) (found
version
>>> >>> "1.5.1")
>>> >>> > Call Stack (most recent call first):
>>> >>> >
>>> >>> >
>>> >>>
>>> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393
>>> >>> > (_FPHSA_FAILURE_MESSAGE)
>>> >>> >   cmake_modules/FindParquet.cmake:116
>>> >>> (find_package_handle_standard_args)
>>> >>> >   CMakeLists.txt:426 (find_package)
>>> >>> >
>>> >>> > Does Parquet need to be installed separately?
>>> >>> >
>>> >>> > With Regards,
>>> >>> > Vibhatha Abeykoon,
>>> >>> > Research Assistant,
>>> >>> > Intelligent Systems Engineering,
>>> >>> > Indiana University Bloomington,
>>> >>> > Cell : +1-812-955-1394
>>> >>> > Web: https://www.vibhatha.org
>>> >>> > <https://www.linkedin.com/in/vibhathaabeykoon/>
>>> >>> >
>>> >>> >
>>> >>> > On Fri, May 1, 2020 at 1:55 PM Wes McKinney <wesmckinn@gmail.com>
>>> >>> wrote:
>>> >>> >
>>> >>> >> Try this instead
>>> >>> >>
>>> >>> >> export
>>> ARROW_HOME=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist
>>> >>> >> export LD_LIBRARY_PATH=$ARROW_HOME/lib:$LD_LIBRARY_PATH
>>> >>> >>
>>> >>> >> Make sure to use
>>> >>> >>
>>> >>> >> -DCMAKE_INSTALL_PREFIX=$ARROW_HOME -DCMAKE_INSTALL_LIBDIR=lib
>>> >>> >>
>>> >>> >> I renamed from "arrowmylibs" to "dist" so that it's less
confusing
>>> >>> >> what that directory represents (it doesn't contains the
libs
>>> >>> >> directory, but rather the directories include/, lib/, etc.)
>>> >>> >>
>>> >>> >>
>>> >>> >> On Fri, May 1, 2020 at 12:51 PM Vibhatha Abeykoon <
>>> vibhatha@gmail.com>
>>> >>> >> wrote:
>>> >>> >> >
>>> >>> >> > I didn't clearly mention the config I used,
>>> >>> >> >
>>> >>> >> > export
>>> >>> >>
>>> ARROW_HOME=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs
>>> >>> >> > export LD_LIBRARY_PATH=$ARROW_HOME:$LD_LIBRARY_PATH
>>> >>> >> >
>>> >>> >> > Before starting I added these ENV vars.
>>> >>> >> >
>>> >>> >> > My objective is to get all the libs inside my build
setting. I
>>> use
>>> >>> arrow
>>> >>> >> as one of the core libraries in my project.
>>> >>> >> > I have to build it in the following way,
>>> >>> >> >
>>> >>> >> > 1. Arrow clone from git
>>> >>> >> > 2. Arrow CPP built
>>> >>> >> > 3. Using arrow CPP I develop custom functions upon
loaded data
>>> >>> >> > 4. Integrate Cython APIs for these custom functions
>>> >>> >> > 5. Use Pyarrow Cython to provide more functionality
from Python
>>> end
>>> >>> to
>>> >>> >> Cython to my CPP lib which uses Arrow.
>>> >>> >> >
>>> >>> >> > This is the build that I am trying to formulate. So
I have to
>>> keep
>>> >>> those
>>> >>> >> libs there, and the idea is to build python from
>>> >>> >> > the same cloned source. The challenge was to keep
all the
>>> shared libs
>>> >>> >> from arrow, my-cython libs to point in the right
>>> >>> >> > direction. Is this a clear description?
>>> >>> >> >
>>> >>> >> > With Regards,
>>> >>> >> > Vibhatha Abeykoon,
>>> >>> >> > Research Assistant,
>>> >>> >> > Intelligent Systems Engineering,
>>> >>> >> > Indiana University Bloomington,
>>> >>> >> > Cell : +1-812-955-1394
>>> >>> >> > Web: https://www.vibhatha.org
>>> >>> >> >
>>> >>> >> >
>>> >>> >> >
>>> >>> >> > On Fri, May 1, 2020 at 1:28 PM Wes McKinney <
>>> wesmckinn@gmail.com>
>>> >>> wrote:
>>> >>> >> >>
>>> >>> >> >> This part doesn't look correct
>>> >>> >> >>
>>> >>> >> >> -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
>>> >>> >> >>
>>> >>> >>
>>> >>>
>>> -DCMAKE_INSTALL_LIBDIR=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs
>>> >>> >> >>
>>> >>> >> >> The usual incantation is
>>> >>> >> >>
>>> >>> >> >> -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
>>> >>> >> >>       -DCMAKE_INSTALL_LIBDIR=lib
>>> >>> >> >>
>>> >>> >> >> The reason we pass -DCMAKE_INSTALL_LIBDIR=lib
is that some
>>> systems
>>> >>> >> >> will install libraries in lib32 or lib64 instead
of just lib
>>> >>> >> >>
>>> >>> >> >>
>>> >>> >> >> On Fri, May 1, 2020 at 10:38 AM Vibhatha Abeykoon
<
>>> >>> vibhatha@gmail.com>
>>> >>> >> wrote:
>>> >>> >> >> >
>>> >>> >> >> > Hi Neal,
>>> >>> >> >> >
>>> >>> >> >> > Yes, I added the flag. But I installed my
libs not to
>>> /usr/lib,
>>> >>> but
>>> >>> >> to a different folder.
>>> >>> >> >> >
>>> >>> >> >> > This is the way I built arrow C++,
>>> >>> >> >> >
>>> >>> >> >> > cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME
\
>>> >>> >> >> >
>>> >>> >>
>>> >>>
>>> -DCMAKE_INSTALL_LIBDIR=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs
>>> >>> >> \
>>> >>> >> >> >       -DARROW_WITH_BZ2=OFF \
>>> >>> >> >> >       -DARROW_WITH_ZLIB=OFF \
>>> >>> >> >> >       -DARROW_WITH_ZSTD=OFF \
>>> >>> >> >> >       -DARROW_WITH_LZ4=OFF \
>>> >>> >> >> >       -DARROW_WITH_SNAPPY=OFF \
>>> >>> >> >> >       -DARROW_WITH_BROTLI=OFF \
>>> >>> >> >> >       -DARROW_PARQUET=ON \
>>> >>> >> >> >       -DARROW_PYTHON=ON \
>>> >>> >> >> >       -DARROW_BUILD_TESTS=ON \
>>> >>> >> >> >
>>> >>> >>
>>> >>>
>>> -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3
>>> >>> >> \
>>> >>> >> >> >       ..
>>> >>> >> >> >
>>> >>> >> >> >
>>> >>> >> >> >
>>> >>> >> >> >
>>> >>> >> >> > With Regards,
>>> >>> >> >> > Vibhatha Abeykoon,
>>> >>> >> >> > Research Assistant,
>>> >>> >> >> > Intelligent Systems Engineering,
>>> >>> >> >> > Indiana University Bloomington,
>>> >>> >> >> > Cell : +1-812-955-1394
>>> >>> >> >> > Web: https://www.vibhatha.org
>>> >>> >> >> >
>>> >>> >> >> >
>>> >>> >> >> >
>>> >>> >> >> > On Fri, May 1, 2020 at 11:29 AM Neal Richardson
<
>>> >>> >> neal.p.richardson@gmail.com> wrote:
>>> >>> >> >> >>
>>> >>> >> >> >> Hi Vibhatha,
>>> >>> >> >> >> Did you build Arrow C++ with -DARROW_PYTHON=ON?
>>> >>> >> >> >>
>>> >>> >> >> >> Neal
>>> >>> >> >> >>
>>> >>> >> >> >> On Fri, May 1, 2020 at 8:24 AM Vibhatha
Abeykoon <
>>> >>> vibhatha@gmail.com>
>>> >>> >> wrote:
>>> >>> >> >> >>>
>>> >>> >> >> >>> Hi,
>>> >>> >> >> >>>
>>> >>> >> >> >>> I am trying to integrate Arrow with
an application that I
>>> am
>>> >>> >> developing. Here I build Arrow from the source (CPP) and
use the
>>> API to
>>> >>> >> develop some custom functions to do a scientific calculation
>>> after data
>>> >>> >> loaded with Arrow table API. On top of this, I develop
a Cython
>>> API to
>>> >>> >> design a python API.
>>> >>> >> >> >>>
>>> >>> >> >> >>> In the current stage, I have a new
necessity where I need
>>> to
>>> >>> >> consume Arrow Cython API for my code.
>>> >>> >> >> >>>
>>> >>> >> >> >>> Here It was hard to link the build
libarrow.so.16 with the
>>> >>> >> libarrow_python.so.16 from the installed pyarrow (separately
from
>>> pip).
>>> >>> >> What I realised was everything has to be built from the
same
>>> source, so
>>> >>> >> that I can install pyarrow from the source in my virtual
>>> environment.
>>> >>> >> >> >>>
>>> >>> >> >> >>> Before going through deeper things,
I started by just
>>> building
>>> >>> from
>>> >>> >> source (CPP) and then moving towards installing pyarrow
from the
>>> >>> source.
>>> >>> >> >> >>>
>>> >>> >> >> >>> I tried to follow the guideline form
here,
>>> >>> >> >> >>>
>>> >>> >> >> >>> https://arrow.apache.org/docs/developers/python.html,
>>> >>> >> >> >>>
>>> >>> >> >> >>> But when I found issues in the python
build, I followed
>>> this
>>> >>> source,
>>> >>> >> >> >>> (but still, I used the clone from
the master, not a
>>> released
>>> >>> >> version)
>>> >>> >> >> >>>
>>> >>> >> >> >>>
>>> >>> https://gist.github.com/heavyinfo/04e1326bb9bed9cecb19c2d603c8d521
>>> >>> >> >> >>>
>>> >>> >> >> >>> My environmental variables are as
follows,
>>> >>> >> >> >>>
>>> >>> >> >> >>> python3 setup.py build_ext --inplace
>>> >>> >> >> >>> running build_ext
>>> >>> >> >> >>> -- Running cmake for pyarrow
>>> >>> >> >> >>> cmake
>>> >>> >>
>>> >>>
>>> -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3
>>> >>> >> -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=off
>>> >>> >> -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off
>>> >>> >> -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=on
>>> >>> >> -DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off
>>> >>> -DPYARROW_BUILD_HDFS=off
>>> >>> >> -DPYARROW_USE_TENSORFLOW=off -DPYARROW_BUNDLE_ARROW_CPP=off
>>> >>> >> -DPYARROW_BUNDLE_BOOST=off -DPYARROW_GENERATE_COVERAGE=off
>>> >>> >> -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_PARQUET_USE_SHARED=on
>>> >>> >> -DCMAKE_BUILD_TYPE=release
>>> >>> /home/vibhatha/sandbox/arrow/repos/arrow/python
>>> >>> >> >> >>> -- System processor: x86_64
>>> >>> >> >> >>> -- Arrow build warning level: PRODUCTION
>>> >>> >> >> >>> Using ld linker
>>> >>> >> >> >>> Configured for RELEASE build (set
with cmake
>>> >>> >> -DCMAKE_BUILD_TYPE={release,debug,...})
>>> >>> >> >> >>> -- Build Type: RELEASE
>>> >>> >> >> >>> -- Build output directory:
>>> >>> >>
>>> >>>
>>> /home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/release
>>> >>> >> >> >>> -- Arrow version: 0.18.0 (HOME:
>>> >>> >> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs)
>>> >>> >> >> >>> -- Arrow SO and ABI version: 18
>>> >>> >> >> >>> -- Arrow full SO version: 18.0.0
>>> >>> >> >> >>> -- Found the Arrow core shared library:
>>> >>> >>
>>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so
>>> >>> >> >> >>> -- Found the Arrow core import library:
>>> >>> >>
>>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so
>>> >>> >> >> >>> -- Found the Arrow core static library:
>>> >>> >>
>>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.a
>>> >>> >> >> >>> CMake Error at
>>> >>> >>
>>> >>>
>>> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146
>>> >>> >> (message):
>>> >>> >> >> >>>   Could NOT find ArrowPython (missing:
>>> ARROW_PYTHON_INCLUDE_DIR)
>>> >>> >> (found
>>> >>> >> >> >>>   version "0.18.0")
>>> >>> >> >> >>> Call Stack (most recent call first):
>>> >>> >> >> >>>
>>> >>> >>
>>> >>>
>>> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393
>>> >>> >> (_FPHSA_FAILURE_MESSAGE)
>>> >>> >> >> >>>   cmake_modules/FindArrowPython.cmake:76
>>> >>> >> (find_package_handle_standard_args)
>>> >>> >> >> >>>   CMakeLists.txt:210 (find_package)
>>> >>> >> >> >>>
>>> >>> >> >> >>>
>>> >>> >> >> >>> -- Configuring incomplete, errors
occurred!
>>> >>> >> >> >>> See also
>>> >>> >>
>>> >>>
>>> "/home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeOutput.log".
>>> >>> >> >> >>> error: command 'cmake' failed with
exit status 1
>>> >>> >> >> >>>
>>> >>> >> >> >>> Maybe I am missing some step and
I am not quite sure what
>>> is the
>>> >>> >> issue.
>>> >>> >> >> >>>
>>> >>> >> >> >>> Any pointers to solve this issue?
>>> >>> >> >> >>>
>>> >>> >> >> >>> With Regards,
>>> >>> >> >> >>> Vibhatha
>>> >>> >> >> >>>
>>> >>> >> >> >>>
>>> >>> >>
>>> >>>
>>> >>
>>>
>> --
>> Vibhatha Abeykoon
>>
>

Mime
View raw message