arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vibhatha Abeykoon <vibha...@gmail.com>
Subject Re: PyArrow building from source issue
Date Fri, 01 May 2020 15:37:54 GMT
Hi Neal,

Yes, I added the flag. But I installed my libs not to /usr/lib, but to a
different folder.

This is the way I built arrow C++,

cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \

-DCMAKE_INSTALL_LIBDIR=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs
\
      -DARROW_WITH_BZ2=OFF \
      -DARROW_WITH_ZLIB=OFF \
      -DARROW_WITH_ZSTD=OFF \
      -DARROW_WITH_LZ4=OFF \
      -DARROW_WITH_SNAPPY=OFF \
      -DARROW_WITH_BROTLI=OFF \
      -DARROW_PARQUET=ON \
      -DARROW_PYTHON=ON \
      -DARROW_BUILD_TESTS=ON \

-DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3
\
      ..




With Regards,
Vibhatha Abeykoon,
Research Assistant,
Intelligent Systems Engineering,
Indiana University Bloomington,
Cell : +1-812-955-1394
Web: https://www.vibhatha.org
<https://www.linkedin.com/in/vibhathaabeykoon/>


On Fri, May 1, 2020 at 11:29 AM Neal Richardson <neal.p.richardson@gmail.com>
wrote:

> Hi Vibhatha,
> Did you build Arrow C++ with -DARROW_PYTHON=ON?
>
> Neal
>
> On Fri, May 1, 2020 at 8:24 AM Vibhatha Abeykoon <vibhatha@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am trying to integrate Arrow with an application that I am developing.
>> Here I build Arrow from the source (CPP) and use the API to develop some
>> custom functions to do a scientific calculation after data loaded with
>> Arrow table API. On top of this, I develop a Cython API to design a python
>> API.
>>
>> In the current stage, I have a new necessity where I need to consume
>> Arrow Cython API for my code.
>>
>> Here It was hard to link the build libarrow.so.16 with the
>> libarrow_python.so.16 from the installed pyarrow (separately from pip).
>> What I realised was everything has to be built from the same source, so
>> that I can install pyarrow from the source in my virtual environment.
>>
>> Before going through deeper things, I started by just building from
>> source (CPP) and then moving towards installing pyarrow from the source.
>>
>> I tried to follow the guideline form here,
>>
>> https://arrow.apache.org/docs/developers/python.html,
>>
>> But when I found issues in the python build, I followed this source,
>> (but still, I used the clone from the master, not a released version)
>>
>> https://gist.github.com/heavyinfo/04e1326bb9bed9cecb19c2d603c8d521
>>
>> My environmental variables are as follows,
>>
>> python3 setup.py build_ext --inplace
>> running build_ext
>> -- Running cmake for pyarrow
>> cmake -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3
 -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=off -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off
-DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=on -DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off
-DPYARROW_BUILD_HDFS=off -DPYARROW_USE_TENSORFLOW=off -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_BOOST=off
-DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_PARQUET_USE_SHARED=on
-DCMAKE_BUILD_TYPE=release /home/vibhatha/sandbox/arrow/repos/arrow/python
>> -- System processor: x86_64
>> -- Arrow build warning level: PRODUCTION
>> Using ld linker
>> Configured for RELEASE build (set with cmake -DCMAKE_BUILD_TYPE={release,debug,...})
>> -- Build Type: RELEASE
>> -- Build output directory: /home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/release
>> -- Arrow version: 0.18.0 (HOME: /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs)
>> -- Arrow SO and ABI version: 18
>> -- Arrow full SO version: 18.0.0
>> -- Found the Arrow core shared library: /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so
>> -- Found the Arrow core import library: /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so
>> -- Found the Arrow core static library: /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.a
>> CMake Error at /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146
(message):
>>   Could NOT find ArrowPython (missing: ARROW_PYTHON_INCLUDE_DIR) (found
>>   version "0.18.0")
>> Call Stack (most recent call first):
>>   /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
>>   cmake_modules/FindArrowPython.cmake:76 (find_package_handle_standard_args)
>>   CMakeLists.txt:210 (find_package)
>>
>>
>> -- Configuring incomplete, errors occurred!
>> See also "/home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeOutput.log".
>> error: command 'cmake' failed with exit status 1
>>
>> Maybe I am missing some step and I am not quite sure what is the issue.
>>
>> Any pointers to solve this issue?
>>
>> With Regards,
>> Vibhatha
>>
>>
>>

Mime
View raw message