arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vibhatha Abeykoon <vibha...@gmail.com>
Subject Re: PyArrow building from source issue
Date Sun, 03 May 2020 00:42:08 GMT
I understand. My main question is the following, building arrow cpp and
pyarrow from the same source would allow seamless integration with another
third party library which uses the arrow cpp and cython API?

On Sat, May 2, 2020 at 8:18 PM Sutou Kouhei <kou@clear-code.com> wrote:

> Hi,
>
> > -DCMAKE_INSTALL_LIBDIR=lib \
> >
> > this path is relative to ARROW_HOME, correct me if I am wrong?
>
> It's relative to CMAKE_INSTALL_PREFIX not ARROW_HOME.
>
> > So with this source build, it doesn't necessarily copy files to /usr/lib
> ?
>
> You don't need to copy.
>
>
> BTW, do you still want to use custom CMAKE_INSTALL_LIBDIR?
> If you want to do, you can't use the ARROW_HOME environment
> variable.
>
> Again, we don't recommend that you use custom
> CMAKE_INSTALL_LIBDIR if you aren't a CMake expert.
>
>
> Thanks,
> --
> kou
>
> In <CABC-QT=0Me4TQcCQbK7uqK1v-0=d0dy=HWyCJjaeU58XTLpP8Q@mail.gmail.com>
>   "Re: PyArrow building from source issue" on Sat, 2 May 2020 19:01:14
> -0400,
>   Vibhatha Abeykoon <vibhatha@gmail.com> wrote:
>
> > One thing I noticed,
> >
> > -DCMAKE_INSTALL_LIBDIR=lib \
> >
> > this path is relative to ARROW_HOME, correct me if I am wrong?
> >
> > So with this source build, it doesn't necessarily copy files to /usr/lib
> ?
> >
> > With Regards,
> > Vibhatha Abeykoon
> >
> >
> > On Sat, May 2, 2020 at 6:12 PM Vibhatha Abeykoon <vibhatha@gmail.com>
> wrote:
> >
> >> So far this is the updated steps,
> >>
> >> pwd => /home/vibhatha/sandbox/arrow/repos
> >>
> >> export
> PYTHON_EXEC=/home/vibhatha/sandbox/arrow/repos/ENVARROW/bin/python3
> >> export LIB_DIR=$(pwd)/libs
> >>
> >> mkdir $LIB_DIR
> >>
> >> ## ENV CONFIGS
> >>
> >> export ARROW_HOME=export ARROW_HOME=$(pwd)/dist
> >> export LD_LIBRARY_PATH=$(pwd)/dist/lib:$LD_LIBRARY_PATH
> >>
> >> ## MAKE BUILD DIRS
> >>
> >> mkdir arrow/cpp/build
> >> pushd arrow/cpp/build
> >>
> >> cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
> >>       -DCMAKE_INSTALL_LIBDIR=$LIB_DIR \
> >>       -DARROW_WITH_BZ2=OFF \
> >>       -DARROW_WITH_ZLIB=OFF \
> >>       -DARROW_WITH_ZSTD=OFF \
> >>       -DARROW_WITH_LZ4=OFF \
> >>       -DARROW_WITH_SNAPPY=OFF \
> >>       -DARROW_WITH_BROTLI=OFF \
> >>       -DARROW_PARQUET=OFF \
> >>       -DARROW_PYTHON=ON \
> >>       -DARROW_BUILD_TESTS=ON \
> >>       -DPYTHON_EXECUTABLE=$PYTHON_EXEC \
> >>       ..
> >>
> >> make -j4
> >> make install
> >>
> >> popd
> >>
> >> export LD_LIBRARY_PATH=$LIB_DIR:$LD_LIBRARY_PATH
> >>
> >> pushd arrow/python
> >> PYARROW_CMAKE_OPTIONS="-DCMAKE_MODULE_PATH="${LIB_DIR}"/cmake/arrow"
> >> python3 setup.py install
> >>
> >>
> >> *Getting an error *
> >>
> >> CMake Error at
> >>
> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146
> >> (message):
> >>   Could NOT find Arrow (missing: ARROW_LIB_DIR) (found version "0.16.0")
> >> Call Stack (most recent call first):
> >>
> >>
> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393
> >> (_FPHSA_FAILURE_MESSAGE)
> >>
>  /home/vibhatha/sandbox/arrow/repos/libs/cmake/arrow/FindArrow.cmake:420
> >> (find_package_handle_standard_args)
> >>
> >>
> /home/vibhatha/sandbox/arrow/repos/libs/cmake/arrow/FindArrowPython.cmake:46
> >> (find_package)
> >>   CMakeLists.txt:204 (find_package)
> >>
> >>
> >> Then, I added the following in FindArrow.cmake
> >>
> >> /home/vibhatha/sandbox/arrow/repos/libs/cmake/arrow/FindArrow.cmake
> >>
> >> set(ARROW_LIB_DIR /home/vibhatha/sandbox/arrow/repos/libs)
> >>
> >> Re-ran the script,
> >>
> >> *Getting Error*
> >>
> >> CMake Error at
> >>
> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146
> >> (message):
> >>   Could NOT find ArrowPython (missing: ARROW_PYTHON_LIB_DIR) (found
> version
> >>   "0.16.0")
> >> Call Stack (most recent call first):
> >>
> >>
> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393
> >> (_FPHSA_FAILURE_MESSAGE)
> >>
> >>
> /home/vibhatha/sandbox/arrow/repos/libs/cmake/arrow/FindArrowPython.cmake:78
> >> (find_package_handle_standard_args)
> >>   CMakeLists.txt:204 (find_package)
> >>
> >> Added the following
> >>
> /home/vibhatha/sandbox/arrow/repos/libs/cmake/arrow/FindArrowPython.cmake
> >>
> >> set(ARROW_PYTHON_LIB_DIR /home/vibhatha/sandbox/arrow/repos/libs)
> >>
> >>
> >> *It seems a path I added could be wrong or a build flag is missing*
> >>
> >> -- Arrow version: 0.16.0 (HOME: /home/vibhatha/sandbox/arrow/repos/dist)
> >> -- Arrow SO and ABI version: 16
> >> -- Arrow full SO version: 16.0.0
> >>
> >>
> >>
> >>
> >>
> >>
> >> *-- Found the Arrow core shared library: ARROW_shared_lib-NOTFOUND--
> Found
> >> the Arrow core import library: -- Found the Arrow core static library:
> >> ARROW_static_lib-NOTFOUND-- Found the Arrow Python by HOME:
> >> /home/vibhatha/sandbox/arrow/repos/dist-- Found the Arrow Python shared
> >> library: ARROW_PYTHON_shared_lib-NOTFOUND-- Found the Arrow Python
> import
> >> library: -- Found the Arrow Python static library:
> >> ARROW_PYTHON_static_lib-NOTFOUND*
> >> -- Configuring done
> >> -- Generating done
> >> -- Build files have been written to:
> >>
> /home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8
> >> -- Finished cmake for pyarrow
> >> -- Running cmake --build for pyarrow
> >> cmake --build . --config release --
> >> [  6%] Compiling Cython CXX source for _compute...
> >> [  6%] Built target _compute_pyx
> >> Scanning dependencies of target _compute
> >> [ 13%] Building CXX object CMakeFiles/_compute.dir/_compute.cpp.o
> >> [ 20%] Linking CXX shared module release/_
> >> compute.cpython-38-x86_64-linux-gnu.so
> >>
> >>
> >> */usr/bin/ld: cannot find -larrow_shared/usr/bin/ld: cannot find
> >> -larrow_python_shared*collect2: error: ld returned 1 exit status
> >> make[2]: *** [CMakeFiles/_compute.dir/build.make:84: release/_
> >> compute.cpython-38-x86_64-linux-gnu.so] Error 1
> >> make[1]: *** [CMakeFiles/Makefile2:89: CMakeFiles/_compute.dir/all]
> Error 2
> >> make: *** [Makefile:84: all] Error 2
> >> error: command 'cmake' failed with exit status 2
> >>
> >> The log also shows that the shared or static libraries have not been
> >> found,
> >>
> >>
> >> With Regards,
> >> Vibhatha Abeykoon
> >>
> >>
> >> On Fri, May 1, 2020 at 8:23 PM Vibhatha Abeykoon <vibhatha@gmail.com>
> >> wrote:
> >>
> >>> Thank you for the clarification.
> >>>
> >>> On Fri, May 1, 2020 at 8:10 PM Sutou Kouhei <kou@clear-code.com>
> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> It'll work.
> >>>> Note that the LD_LIBRARY_PATH is the environment variable at
> >>>> run time. You need to specify the correct ARROW_HOME or
> >>>> -DCMAKE_MODULE_PATH at build time too.
> >>>>
> >>>>
> >>>> Thanks,
> >>>> --
> >>>> kou
> >>>>
> >>>> In <CABC-QT=
> U_+AE8LaMmHwfhJP+4wvVKJ0Y-z7styK0TxtrJyZdNQ@mail.gmail.com>
> >>>>   "Re: PyArrow building from source issue" on Fri, 1 May 2020 18:13:54
> >>>> -0400,
> >>>>   Vibhatha Abeykoon <vibhatha@gmail.com> wrote:
> >>>>
> >>>> > I will elaborate a couple of reasons,
> >>>> >
> >>>> > When there are a couple of versions of Arrow, used for different
> >>>> projects
> >>>> > depending on various development choices, it is convenient for
me to
> >>>> keep
> >>>> > them pointed towards a folder of my choice.
> >>>> > Then refer to it and continue the work. Correct me if I am wrong,
> what
> >>>> if I
> >>>> > point to this folder of my choice and add it to the LD_LIBRARY_PATH.
> >>>> > Will this cause issues?
> >>>> >
> >>>> > With Regards,
> >>>> > Vibhatha Abeykoon
> >>>> >
> >>>> >
> >>>> > On Fri, May 1, 2020 at 6:09 PM Vibhatha Abeykoon <
> vibhatha@gmail.com>
> >>>> wrote:
> >>>> >
> >>>> >> Okay, thank you for your response.
> >>>> >>
> >>>> >> With Regards,
> >>>> >> Vibhatha Abeykoon
> >>>> >>
> >>>> >>
> >>>> >> On Fri, May 1, 2020 at 5:17 PM Sutou Kouhei <kou@clear-code.com>
> >>>> wrote:
> >>>> >>
> >>>> >>> Hi,
> >>>> >>>
> >>>> >>> > I used these settings, I still want the libs to be
in a custom
> >>>> >>> directory,
> >>>> >>> > not in lib/
> >>>> >>>
> >>>> >>> Why? We don't recommend it if you don't a CMake expert.
> >>>> >>>
> >>>> >>> You can't do this if you use ARROW_HOME environment
> >>>> >>> variable.
> >>>> >>>
> >>>> >>> You may able to do this by removing ARROW_HOME environment
> >>>> >>> variable and adding
> >>>> >>>
> >>>> >>>
> >>>>
> PYARROW_CMAKE_OPTIONS="-DCMAKE_MODULE_PATH=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist/lib/cmake/arrow"
> >>>> >>> environment variable or something.
> >>>> >>>
> >>>> >>>
> >>>> >>> Thanks,
> >>>> >>> --
> >>>> >>> kou
> >>>> >>>
> >>>> >>> In <CABC-QTmsekZwjWG4X=
> >>>> aWrfQZgYi+qLdyEc_kvbN2njgtwJLofw@mail.gmail.com>
> >>>> >>>   "Re: PyArrow building from source issue" on Fri, 1 May
2020
> >>>> 15:05:51
> >>>> >>> -0400,
> >>>> >>>   Vibhatha Abeykoon <vibhatha@gmail.com> wrote:
> >>>> >>>
> >>>> >>> > export
> ARROW_HOME=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist
> >>>> >>> > export LD_LIBRARY_PATH=$ARROW_HOME/lib:$LD_LIBRARY_PATH
> >>>> >>> >
> >>>> >>> > export PYARROW_WITH_PARQUET=1
> >>>> >>> > export PYARROW_WITH_PYTHON=1
> >>>> >>> > export PYARROW_WITH_BUILD_TESTS=1
> >>>> >>> >
> >>>> >>> > cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
> >>>> >>> >
> >>>> >>> >
> >>>> >>>
> >>>>
> -DCMAKE_INSTALL_LIBDIR=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs
> >>>> >>> > \
> >>>> >>> >       -DARROW_WITH_BZ2=OFF \
> >>>> >>> >       -DARROW_WITH_ZLIB=OFF \
> >>>> >>> >       -DARROW_WITH_ZSTD=OFF \
> >>>> >>> >       -DARROW_WITH_LZ4=OFF \
> >>>> >>> >       -DARROW_WITH_SNAPPY=OFF \
> >>>> >>> >       -DARROW_WITH_BROTLI=OFF \
> >>>> >>> >       -DARROW_PARQUET=ON \
> >>>> >>> >       -DARROW_PYTHON=ON \
> >>>> >>> >       -DARROW_BUILD_TESTS=ON \
> >>>> >>> >
> >>>> >>> >
> >>>> >>>
> >>>>
> -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3
> >>>> >>> > \
> >>>> >>> >       ..
> >>>> >>> >
> >>>> >>> >
> >>>> >>> > I used these settings, I still want the libs to be
in a custom
> >>>> >>> directory,
> >>>> >>> > not in lib/
> >>>> >>> >
> >>>> >>> > Does it make things not work?
> >>>> >>> >
> >>>> >>> > Now I get the following error,
> >>>> >>> >
> >>>> >>> > python setup.py build_ext --inplace
> >>>> >>> > WARNING: The wheel package is not available.
> >>>> >>> > running build_ext
> >>>> >>> > -- Running cmake for pyarrow
> >>>> >>> > cmake
> >>>> >>>
> -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/ENVARROW/bin/python
> >>>> >>> >  -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=off
> >>>> >>> > -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off
> >>>> >>> > -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=on
> >>>> >>> > -DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off
> >>>> >>> -DPYARROW_BUILD_HDFS=off
> >>>> >>> > -DPYARROW_USE_TENSORFLOW=off -DPYARROW_BUNDLE_ARROW_CPP=off
> >>>> >>> > -DPYARROW_BUNDLE_BOOST=off -DPYARROW_GENERATE_COVERAGE=off
> >>>> >>> > -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_PARQUET_USE_SHARED=on
> >>>> >>> > -DCMAKE_BUILD_TYPE=release
> >>>> >>> /home/vibhatha/sandbox/arrow/repos/arrow/python
> >>>> >>> > -- System processor: x86_64
> >>>> >>> > -- Arrow build warning level: PRODUCTION
> >>>> >>> > Using ld linker
> >>>> >>> > Configured for RELEASE build (set with cmake
> >>>> >>> > -DCMAKE_BUILD_TYPE={release,debug,...})
> >>>> >>> > -- Build Type: RELEASE
> >>>> >>> > -- Build output directory:
> >>>> >>> >
> >>>> >>>
> >>>>
> /home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/release
> >>>> >>> > -- Arrow version: 0.18.0 (HOME:
> >>>> >>> > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist)
> >>>> >>> > -- Arrow SO and ABI version: 18
> >>>> >>> > -- Arrow full SO version: 18.0.0
> >>>> >>> > -- Found the Arrow core shared library:
> >>>> >>> >
> >>>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so
> >>>> >>> > -- Found the Arrow core import library:
> >>>> >>> >
> >>>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so
> >>>> >>> > -- Found the Arrow core static library:
> >>>> >>> >
> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.a
> >>>> >>> > -- Found the Arrow Python by HOME:
> >>>> >>> > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist
> >>>> >>> > -- Found the Arrow Python shared library:
> >>>> >>> >
> >>>> >>>
> >>>>
> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow_python.so
> >>>> >>> > -- Found the Arrow Python import library:
> >>>> >>> >
> >>>> >>>
> >>>>
> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow_python.so
> >>>> >>> > -- Found the Arrow Python static library:
> >>>> >>> >
> >>>> >>>
> >>>>
> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow_python.a
> >>>> >>> > CMake Error at
> >>>> >>> >
> >>>> >>>
> >>>>
> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146
> >>>> >>> > (message):
> >>>> >>> >   Could NOT find Parquet (missing: PARQUET_LIB_DIR)
(found
> version
> >>>> >>> "1.5.1")
> >>>> >>> > Call Stack (most recent call first):
> >>>> >>> >
> >>>> >>> >
> >>>> >>>
> >>>>
> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393
> >>>> >>> > (_FPHSA_FAILURE_MESSAGE)
> >>>> >>> >   cmake_modules/FindParquet.cmake:116
> >>>> >>> (find_package_handle_standard_args)
> >>>> >>> >   CMakeLists.txt:426 (find_package)
> >>>> >>> >
> >>>> >>> > Does Parquet need to be installed separately?
> >>>> >>> >
> >>>> >>> > With Regards,
> >>>> >>> > Vibhatha Abeykoon,
> >>>> >>> > Research Assistant,
> >>>> >>> > Intelligent Systems Engineering,
> >>>> >>> > Indiana University Bloomington,
> >>>> >>> > Cell : +1-812-955-1394
> >>>> >>> > Web: https://www.vibhatha.org
> >>>> >>> > <https://www.linkedin.com/in/vibhathaabeykoon/>
> >>>> >>> >
> >>>> >>> >
> >>>> >>> > On Fri, May 1, 2020 at 1:55 PM Wes McKinney <
> wesmckinn@gmail.com>
> >>>> >>> wrote:
> >>>> >>> >
> >>>> >>> >> Try this instead
> >>>> >>> >>
> >>>> >>> >> export
> >>>> ARROW_HOME=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist
> >>>> >>> >> export LD_LIBRARY_PATH=$ARROW_HOME/lib:$LD_LIBRARY_PATH
> >>>> >>> >>
> >>>> >>> >> Make sure to use
> >>>> >>> >>
> >>>> >>> >> -DCMAKE_INSTALL_PREFIX=$ARROW_HOME -DCMAKE_INSTALL_LIBDIR=lib
> >>>> >>> >>
> >>>> >>> >> I renamed from "arrowmylibs" to "dist" so that
it's less
> confusing
> >>>> >>> >> what that directory represents (it doesn't contains
the libs
> >>>> >>> >> directory, but rather the directories include/,
lib/, etc.)
> >>>> >>> >>
> >>>> >>> >>
> >>>> >>> >> On Fri, May 1, 2020 at 12:51 PM Vibhatha Abeykoon
<
> >>>> vibhatha@gmail.com>
> >>>> >>> >> wrote:
> >>>> >>> >> >
> >>>> >>> >> > I didn't clearly mention the config I used,
> >>>> >>> >> >
> >>>> >>> >> > export
> >>>> >>> >>
> >>>> ARROW_HOME=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs
> >>>> >>> >> > export LD_LIBRARY_PATH=$ARROW_HOME:$LD_LIBRARY_PATH
> >>>> >>> >> >
> >>>> >>> >> > Before starting I added these ENV vars.
> >>>> >>> >> >
> >>>> >>> >> > My objective is to get all the libs inside
my build setting.
> I
> >>>> use
> >>>> >>> arrow
> >>>> >>> >> as one of the core libraries in my project.
> >>>> >>> >> > I have to build it in the following way,
> >>>> >>> >> >
> >>>> >>> >> > 1. Arrow clone from git
> >>>> >>> >> > 2. Arrow CPP built
> >>>> >>> >> > 3. Using arrow CPP I develop custom functions
upon loaded
> data
> >>>> >>> >> > 4. Integrate Cython APIs for these custom
functions
> >>>> >>> >> > 5. Use Pyarrow Cython to provide more functionality
from
> Python
> >>>> end
> >>>> >>> to
> >>>> >>> >> Cython to my CPP lib which uses Arrow.
> >>>> >>> >> >
> >>>> >>> >> > This is the build that I am trying to formulate.
So I have to
> >>>> keep
> >>>> >>> those
> >>>> >>> >> libs there, and the idea is to build python from
> >>>> >>> >> > the same cloned source. The challenge was
to keep all the
> >>>> shared libs
> >>>> >>> >> from arrow, my-cython libs to point in the right
> >>>> >>> >> > direction. Is this a clear description?
> >>>> >>> >> >
> >>>> >>> >> > With Regards,
> >>>> >>> >> > Vibhatha Abeykoon,
> >>>> >>> >> > Research Assistant,
> >>>> >>> >> > Intelligent Systems Engineering,
> >>>> >>> >> > Indiana University Bloomington,
> >>>> >>> >> > Cell : +1-812-955-1394
> >>>> >>> >> > Web: https://www.vibhatha.org
> >>>> >>> >> >
> >>>> >>> >> >
> >>>> >>> >> >
> >>>> >>> >> > On Fri, May 1, 2020 at 1:28 PM Wes McKinney
<
> >>>> wesmckinn@gmail.com>
> >>>> >>> wrote:
> >>>> >>> >> >>
> >>>> >>> >> >> This part doesn't look correct
> >>>> >>> >> >>
> >>>> >>> >> >> -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
> >>>> >>> >> >>
> >>>> >>> >>
> >>>> >>>
> >>>>
> -DCMAKE_INSTALL_LIBDIR=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs
> >>>> >>> >> >>
> >>>> >>> >> >> The usual incantation is
> >>>> >>> >> >>
> >>>> >>> >> >> -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
> >>>> >>> >> >>       -DCMAKE_INSTALL_LIBDIR=lib
> >>>> >>> >> >>
> >>>> >>> >> >> The reason we pass -DCMAKE_INSTALL_LIBDIR=lib
is that some
> >>>> systems
> >>>> >>> >> >> will install libraries in lib32 or lib64
instead of just lib
> >>>> >>> >> >>
> >>>> >>> >> >>
> >>>> >>> >> >> On Fri, May 1, 2020 at 10:38 AM Vibhatha
Abeykoon <
> >>>> >>> vibhatha@gmail.com>
> >>>> >>> >> wrote:
> >>>> >>> >> >> >
> >>>> >>> >> >> > Hi Neal,
> >>>> >>> >> >> >
> >>>> >>> >> >> > Yes, I added the flag. But I installed
my libs not to
> >>>> /usr/lib,
> >>>> >>> but
> >>>> >>> >> to a different folder.
> >>>> >>> >> >> >
> >>>> >>> >> >> > This is the way I built arrow C++,
> >>>> >>> >> >> >
> >>>> >>> >> >> > cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME
\
> >>>> >>> >> >> >
> >>>> >>> >>
> >>>> >>>
> >>>>
> -DCMAKE_INSTALL_LIBDIR=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs
> >>>> >>> >> \
> >>>> >>> >> >> >       -DARROW_WITH_BZ2=OFF \
> >>>> >>> >> >> >       -DARROW_WITH_ZLIB=OFF \
> >>>> >>> >> >> >       -DARROW_WITH_ZSTD=OFF \
> >>>> >>> >> >> >       -DARROW_WITH_LZ4=OFF \
> >>>> >>> >> >> >       -DARROW_WITH_SNAPPY=OFF \
> >>>> >>> >> >> >       -DARROW_WITH_BROTLI=OFF \
> >>>> >>> >> >> >       -DARROW_PARQUET=ON \
> >>>> >>> >> >> >       -DARROW_PYTHON=ON \
> >>>> >>> >> >> >       -DARROW_BUILD_TESTS=ON \
> >>>> >>> >> >> >
> >>>> >>> >>
> >>>> >>>
> >>>>
> -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3
> >>>> >>> >> \
> >>>> >>> >> >> >       ..
> >>>> >>> >> >> >
> >>>> >>> >> >> >
> >>>> >>> >> >> >
> >>>> >>> >> >> >
> >>>> >>> >> >> > With Regards,
> >>>> >>> >> >> > Vibhatha Abeykoon,
> >>>> >>> >> >> > Research Assistant,
> >>>> >>> >> >> > Intelligent Systems Engineering,
> >>>> >>> >> >> > Indiana University Bloomington,
> >>>> >>> >> >> > Cell : +1-812-955-1394
> >>>> >>> >> >> > Web: https://www.vibhatha.org
> >>>> >>> >> >> >
> >>>> >>> >> >> >
> >>>> >>> >> >> >
> >>>> >>> >> >> > On Fri, May 1, 2020 at 11:29 AM
Neal Richardson <
> >>>> >>> >> neal.p.richardson@gmail.com> wrote:
> >>>> >>> >> >> >>
> >>>> >>> >> >> >> Hi Vibhatha,
> >>>> >>> >> >> >> Did you build Arrow C++ with
-DARROW_PYTHON=ON?
> >>>> >>> >> >> >>
> >>>> >>> >> >> >> Neal
> >>>> >>> >> >> >>
> >>>> >>> >> >> >> On Fri, May 1, 2020 at 8:24
AM Vibhatha Abeykoon <
> >>>> >>> vibhatha@gmail.com>
> >>>> >>> >> wrote:
> >>>> >>> >> >> >>>
> >>>> >>> >> >> >>> Hi,
> >>>> >>> >> >> >>>
> >>>> >>> >> >> >>> I am trying to integrate
Arrow with an application that
> I
> >>>> am
> >>>> >>> >> developing. Here I build Arrow from the source
(CPP) and use
> the
> >>>> API to
> >>>> >>> >> develop some custom functions to do a scientific
calculation
> >>>> after data
> >>>> >>> >> loaded with Arrow table API. On top of this, I
develop a Cython
> >>>> API to
> >>>> >>> >> design a python API.
> >>>> >>> >> >> >>>
> >>>> >>> >> >> >>> In the current stage, I
have a new necessity where I
> need
> >>>> to
> >>>> >>> >> consume Arrow Cython API for my code.
> >>>> >>> >> >> >>>
> >>>> >>> >> >> >>> Here It was hard to link
the build libarrow.so.16 with
> the
> >>>> >>> >> libarrow_python.so.16 from the installed pyarrow
(separately
> from
> >>>> pip).
> >>>> >>> >> What I realised was everything has to be built
from the same
> >>>> source, so
> >>>> >>> >> that I can install pyarrow from the source in
my virtual
> >>>> environment.
> >>>> >>> >> >> >>>
> >>>> >>> >> >> >>> Before going through deeper
things, I started by just
> >>>> building
> >>>> >>> from
> >>>> >>> >> source (CPP) and then moving towards installing
pyarrow from
> the
> >>>> >>> source.
> >>>> >>> >> >> >>>
> >>>> >>> >> >> >>> I tried to follow the guideline
form here,
> >>>> >>> >> >> >>>
> >>>> >>> >> >> >>> https://arrow.apache.org/docs/developers/python.html,
> >>>> >>> >> >> >>>
> >>>> >>> >> >> >>> But when I found issues
in the python build, I followed
> >>>> this
> >>>> >>> source,
> >>>> >>> >> >> >>> (but still, I used the clone
from the master, not a
> >>>> released
> >>>> >>> >> version)
> >>>> >>> >> >> >>>
> >>>> >>> >> >> >>>
> >>>> >>>
> https://gist.github.com/heavyinfo/04e1326bb9bed9cecb19c2d603c8d521
> >>>> >>> >> >> >>>
> >>>> >>> >> >> >>> My environmental variables
are as follows,
> >>>> >>> >> >> >>>
> >>>> >>> >> >> >>> python3 setup.py build_ext
--inplace
> >>>> >>> >> >> >>> running build_ext
> >>>> >>> >> >> >>> -- Running cmake for pyarrow
> >>>> >>> >> >> >>> cmake
> >>>> >>> >>
> >>>> >>>
> >>>>
> -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3
> >>>> >>> >> -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=off
> >>>> >>> >> -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off
> >>>> >>> >> -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=on
> >>>> >>> >> -DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off
> >>>> >>> -DPYARROW_BUILD_HDFS=off
> >>>> >>> >> -DPYARROW_USE_TENSORFLOW=off -DPYARROW_BUNDLE_ARROW_CPP=off
> >>>> >>> >> -DPYARROW_BUNDLE_BOOST=off -DPYARROW_GENERATE_COVERAGE=off
> >>>> >>> >> -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_PARQUET_USE_SHARED=on
> >>>> >>> >> -DCMAKE_BUILD_TYPE=release
> >>>> >>> /home/vibhatha/sandbox/arrow/repos/arrow/python
> >>>> >>> >> >> >>> -- System processor: x86_64
> >>>> >>> >> >> >>> -- Arrow build warning level:
PRODUCTION
> >>>> >>> >> >> >>> Using ld linker
> >>>> >>> >> >> >>> Configured for RELEASE build
(set with cmake
> >>>> >>> >> -DCMAKE_BUILD_TYPE={release,debug,...})
> >>>> >>> >> >> >>> -- Build Type: RELEASE
> >>>> >>> >> >> >>> -- Build output directory:
> >>>> >>> >>
> >>>> >>>
> >>>>
> /home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/release
> >>>> >>> >> >> >>> -- Arrow version: 0.18.0
(HOME:
> >>>> >>> >> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs)
> >>>> >>> >> >> >>> -- Arrow SO and ABI version:
18
> >>>> >>> >> >> >>> -- Arrow full SO version:
18.0.0
> >>>> >>> >> >> >>> -- Found the Arrow core
shared library:
> >>>> >>> >>
> >>>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so
> >>>> >>> >> >> >>> -- Found the Arrow core
import library:
> >>>> >>> >>
> >>>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so
> >>>> >>> >> >> >>> -- Found the Arrow core
static library:
> >>>> >>> >>
> >>>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.a
> >>>> >>> >> >> >>> CMake Error at
> >>>> >>> >>
> >>>> >>>
> >>>>
> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146
> >>>> >>> >> (message):
> >>>> >>> >> >> >>>   Could NOT find ArrowPython
(missing:
> >>>> ARROW_PYTHON_INCLUDE_DIR)
> >>>> >>> >> (found
> >>>> >>> >> >> >>>   version "0.18.0")
> >>>> >>> >> >> >>> Call Stack (most recent
call first):
> >>>> >>> >> >> >>>
> >>>> >>> >>
> >>>> >>>
> >>>>
> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393
> >>>> >>> >> (_FPHSA_FAILURE_MESSAGE)
> >>>> >>> >> >> >>>   cmake_modules/FindArrowPython.cmake:76
> >>>> >>> >> (find_package_handle_standard_args)
> >>>> >>> >> >> >>>   CMakeLists.txt:210 (find_package)
> >>>> >>> >> >> >>>
> >>>> >>> >> >> >>>
> >>>> >>> >> >> >>> -- Configuring incomplete,
errors occurred!
> >>>> >>> >> >> >>> See also
> >>>> >>> >>
> >>>> >>>
> >>>>
> "/home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeOutput.log".
> >>>> >>> >> >> >>> error: command 'cmake' failed
with exit status 1
> >>>> >>> >> >> >>>
> >>>> >>> >> >> >>> Maybe I am missing some
step and I am not quite sure
> what
> >>>> is the
> >>>> >>> >> issue.
> >>>> >>> >> >> >>>
> >>>> >>> >> >> >>> Any pointers to solve this
issue?
> >>>> >>> >> >> >>>
> >>>> >>> >> >> >>> With Regards,
> >>>> >>> >> >> >>> Vibhatha
> >>>> >>> >> >> >>>
> >>>> >>> >> >> >>>
> >>>> >>> >>
> >>>> >>>
> >>>> >>
> >>>>
> >>> --
> >>> Vibhatha Abeykoon
> >>>
> >>
>
-- 
Vibhatha Abeykoon

Mime
View raw message