From user-return-421-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Fri May 1 22:14:12 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id BDE1E180634 for ; Sat, 2 May 2020 00:14:11 +0200 (CEST) Received: (qmail 67095 invoked by uid 500); 1 May 2020 22:14:11 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 67085 invoked by uid 99); 1 May 2020 22:14:11 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2020 22:14:11 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 94168C1C04 for ; Fri, 1 May 2020 22:14:10 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.001 X-Spam-Level: X-Spam-Status: No, score=0.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 3Uq39ycSQOC4 for ; Fri, 1 May 2020 22:14:08 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::b41; helo=mail-yb1-xb41.google.com; envelope-from=vibhatha@gmail.com; receiver= Received: from mail-yb1-xb41.google.com (mail-yb1-xb41.google.com [IPv6:2607:f8b0:4864:20::b41]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 778C17DC65 for ; Fri, 1 May 2020 22:14:07 +0000 (UTC) Received: by mail-yb1-xb41.google.com with SMTP id o198so5664553ybg.10 for ; Fri, 01 May 2020 15:14:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=XEvSHhkYnsoW1tyqYHVSVVx75ShfiNCsYP090WYN3ws=; b=GRszXUhvHUhz8oM6YqKodgU7KFTA3hzrJDI0t19twYyR/eMAi7rMoofRnw5vS6Zwwl 0kHUfdIun3Y+0Z064BvLgTAwDlC8qwes9d01BT0x/BnjEE+C1wC82+3a1+7iXmKMNpTS exmJSuf3nAqa+zxIUukiSiiioLMBhpwzSOHojlwKzXiHXOGzzdzpLdkJAzJDXwnEGYxf FbZEYwO6M6KJsma8YnySdGdrZQae0CwcRbUs1ALu3s88LloKlsB8h7I6aCVPecJjUIE8 m7fgEvCpmswBu8+afnbTBECLvQ3ps55rggbxAuXfbDfFLR5sqjHDHQlWkDq6wUhvnD+i LVug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=XEvSHhkYnsoW1tyqYHVSVVx75ShfiNCsYP090WYN3ws=; b=N48rKulH9zWzHaaXw7Z7w2iAoLfdAcahr9AGt2Dr4ZS3rYp4Zwm6/1uqTu7KiYOO7v Vbh49VCxsWjLzMEB9mlstdv+2UYa6UZshBjq6EaUwnZISh5LIRmPcT8jfmzD/Vvxdndp ZbVU/rmWNKgZnjIua49eyu4sOMQjzDWOWBEExJ3wgHBd+Zp64Fvbp5+oI6Mc3JIO5mRK CAGFBdHejG93Viy0Xp1v4AenWi6kXL7WzS4qNrx/NusjBQEaEynxcHaWuG0oejGHNjx+ VdA896bYbnchPDcRsj3mhTL6agyhICU/STSykO93XtnVo5UDAF/7WxkBKyk51MXzFurw 5Ujg== X-Gm-Message-State: AGi0PuaXoMdTFRktao6CdmhQ2JEYkwH634HETVLi2whRK3dM7lsEod7Z UwNqKmz82Je+T2iRpDUvzKgbDnabwhZ2X/YS13QtafsK X-Google-Smtp-Source: APiQypJvKFVU2Zub++gDKuRB+MZGNvlWSqFpN8H1rNJEdK0WadQJt93JaP98ANNuH04XW1/ylceXUfKTo7MbLTpZ5Cg= X-Received: by 2002:a25:b841:: with SMTP id b1mr9858987ybm.492.1588371245800; Fri, 01 May 2020 15:14:05 -0700 (PDT) MIME-Version: 1.0 References: <20200502.061648.1991462595896728267.kou@clear-code.com> In-Reply-To: From: Vibhatha Abeykoon Date: Fri, 1 May 2020 18:13:54 -0400 Message-ID: Subject: Re: PyArrow building from source issue To: user@arrow.apache.org Content-Type: multipart/alternative; boundary="000000000000e09fe405a49d7f67" --000000000000e09fe405a49d7f67 Content-Type: text/plain; charset="UTF-8" I will elaborate a couple of reasons, When there are a couple of versions of Arrow, used for different projects depending on various development choices, it is convenient for me to keep them pointed towards a folder of my choice. Then refer to it and continue the work. Correct me if I am wrong, what if I point to this folder of my choice and add it to the LD_LIBRARY_PATH. Will this cause issues? With Regards, Vibhatha Abeykoon On Fri, May 1, 2020 at 6:09 PM Vibhatha Abeykoon wrote: > Okay, thank you for your response. > > With Regards, > Vibhatha Abeykoon > > > On Fri, May 1, 2020 at 5:17 PM Sutou Kouhei wrote: > >> Hi, >> >> > I used these settings, I still want the libs to be in a custom >> directory, >> > not in lib/ >> >> Why? We don't recommend it if you don't a CMake expert. >> >> You can't do this if you use ARROW_HOME environment >> variable. >> >> You may able to do this by removing ARROW_HOME environment >> variable and adding >> >> PYARROW_CMAKE_OPTIONS="-DCMAKE_MODULE_PATH=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist/lib/cmake/arrow" >> environment variable or something. >> >> >> Thanks, >> -- >> kou >> >> In >> "Re: PyArrow building from source issue" on Fri, 1 May 2020 15:05:51 >> -0400, >> Vibhatha Abeykoon wrote: >> >> > export ARROW_HOME=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist >> > export LD_LIBRARY_PATH=$ARROW_HOME/lib:$LD_LIBRARY_PATH >> > >> > export PYARROW_WITH_PARQUET=1 >> > export PYARROW_WITH_PYTHON=1 >> > export PYARROW_WITH_BUILD_TESTS=1 >> > >> > cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ >> > >> > >> -DCMAKE_INSTALL_LIBDIR=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs >> > \ >> > -DARROW_WITH_BZ2=OFF \ >> > -DARROW_WITH_ZLIB=OFF \ >> > -DARROW_WITH_ZSTD=OFF \ >> > -DARROW_WITH_LZ4=OFF \ >> > -DARROW_WITH_SNAPPY=OFF \ >> > -DARROW_WITH_BROTLI=OFF \ >> > -DARROW_PARQUET=ON \ >> > -DARROW_PYTHON=ON \ >> > -DARROW_BUILD_TESTS=ON \ >> > >> > >> -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3 >> > \ >> > .. >> > >> > >> > I used these settings, I still want the libs to be in a custom >> directory, >> > not in lib/ >> > >> > Does it make things not work? >> > >> > Now I get the following error, >> > >> > python setup.py build_ext --inplace >> > WARNING: The wheel package is not available. >> > running build_ext >> > -- Running cmake for pyarrow >> > cmake >> -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/ENVARROW/bin/python >> > -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=off >> > -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off >> > -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=on >> > -DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off >> -DPYARROW_BUILD_HDFS=off >> > -DPYARROW_USE_TENSORFLOW=off -DPYARROW_BUNDLE_ARROW_CPP=off >> > -DPYARROW_BUNDLE_BOOST=off -DPYARROW_GENERATE_COVERAGE=off >> > -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_PARQUET_USE_SHARED=on >> > -DCMAKE_BUILD_TYPE=release >> /home/vibhatha/sandbox/arrow/repos/arrow/python >> > -- System processor: x86_64 >> > -- Arrow build warning level: PRODUCTION >> > Using ld linker >> > Configured for RELEASE build (set with cmake >> > -DCMAKE_BUILD_TYPE={release,debug,...}) >> > -- Build Type: RELEASE >> > -- Build output directory: >> > >> /home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/release >> > -- Arrow version: 0.18.0 (HOME: >> > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist) >> > -- Arrow SO and ABI version: 18 >> > -- Arrow full SO version: 18.0.0 >> > -- Found the Arrow core shared library: >> > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so >> > -- Found the Arrow core import library: >> > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so >> > -- Found the Arrow core static library: >> > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.a >> > -- Found the Arrow Python by HOME: >> > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist >> > -- Found the Arrow Python shared library: >> > >> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow_python.so >> > -- Found the Arrow Python import library: >> > >> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow_python.so >> > -- Found the Arrow Python static library: >> > >> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow_python.a >> > CMake Error at >> > >> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 >> > (message): >> > Could NOT find Parquet (missing: PARQUET_LIB_DIR) (found version >> "1.5.1") >> > Call Stack (most recent call first): >> > >> > >> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 >> > (_FPHSA_FAILURE_MESSAGE) >> > cmake_modules/FindParquet.cmake:116 >> (find_package_handle_standard_args) >> > CMakeLists.txt:426 (find_package) >> > >> > Does Parquet need to be installed separately? >> > >> > With Regards, >> > Vibhatha Abeykoon, >> > Research Assistant, >> > Intelligent Systems Engineering, >> > Indiana University Bloomington, >> > Cell : +1-812-955-1394 >> > Web: https://www.vibhatha.org >> > >> > >> > >> > On Fri, May 1, 2020 at 1:55 PM Wes McKinney >> wrote: >> > >> >> Try this instead >> >> >> >> export ARROW_HOME=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist >> >> export LD_LIBRARY_PATH=$ARROW_HOME/lib:$LD_LIBRARY_PATH >> >> >> >> Make sure to use >> >> >> >> -DCMAKE_INSTALL_PREFIX=$ARROW_HOME -DCMAKE_INSTALL_LIBDIR=lib >> >> >> >> I renamed from "arrowmylibs" to "dist" so that it's less confusing >> >> what that directory represents (it doesn't contains the libs >> >> directory, but rather the directories include/, lib/, etc.) >> >> >> >> >> >> On Fri, May 1, 2020 at 12:51 PM Vibhatha Abeykoon >> >> wrote: >> >> > >> >> > I didn't clearly mention the config I used, >> >> > >> >> > export >> >> ARROW_HOME=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs >> >> > export LD_LIBRARY_PATH=$ARROW_HOME:$LD_LIBRARY_PATH >> >> > >> >> > Before starting I added these ENV vars. >> >> > >> >> > My objective is to get all the libs inside my build setting. I use >> arrow >> >> as one of the core libraries in my project. >> >> > I have to build it in the following way, >> >> > >> >> > 1. Arrow clone from git >> >> > 2. Arrow CPP built >> >> > 3. Using arrow CPP I develop custom functions upon loaded data >> >> > 4. Integrate Cython APIs for these custom functions >> >> > 5. Use Pyarrow Cython to provide more functionality from Python end >> to >> >> Cython to my CPP lib which uses Arrow. >> >> > >> >> > This is the build that I am trying to formulate. So I have to keep >> those >> >> libs there, and the idea is to build python from >> >> > the same cloned source. The challenge was to keep all the shared libs >> >> from arrow, my-cython libs to point in the right >> >> > direction. Is this a clear description? >> >> > >> >> > With Regards, >> >> > Vibhatha Abeykoon, >> >> > Research Assistant, >> >> > Intelligent Systems Engineering, >> >> > Indiana University Bloomington, >> >> > Cell : +1-812-955-1394 >> >> > Web: https://www.vibhatha.org >> >> > >> >> > >> >> > >> >> > On Fri, May 1, 2020 at 1:28 PM Wes McKinney >> wrote: >> >> >> >> >> >> This part doesn't look correct >> >> >> >> >> >> -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ >> >> >> >> >> >> -DCMAKE_INSTALL_LIBDIR=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs >> >> >> >> >> >> The usual incantation is >> >> >> >> >> >> -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ >> >> >> -DCMAKE_INSTALL_LIBDIR=lib >> >> >> >> >> >> The reason we pass -DCMAKE_INSTALL_LIBDIR=lib is that some systems >> >> >> will install libraries in lib32 or lib64 instead of just lib >> >> >> >> >> >> >> >> >> On Fri, May 1, 2020 at 10:38 AM Vibhatha Abeykoon < >> vibhatha@gmail.com> >> >> wrote: >> >> >> > >> >> >> > Hi Neal, >> >> >> > >> >> >> > Yes, I added the flag. But I installed my libs not to /usr/lib, >> but >> >> to a different folder. >> >> >> > >> >> >> > This is the way I built arrow C++, >> >> >> > >> >> >> > cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ >> >> >> > >> >> >> -DCMAKE_INSTALL_LIBDIR=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs >> >> \ >> >> >> > -DARROW_WITH_BZ2=OFF \ >> >> >> > -DARROW_WITH_ZLIB=OFF \ >> >> >> > -DARROW_WITH_ZSTD=OFF \ >> >> >> > -DARROW_WITH_LZ4=OFF \ >> >> >> > -DARROW_WITH_SNAPPY=OFF \ >> >> >> > -DARROW_WITH_BROTLI=OFF \ >> >> >> > -DARROW_PARQUET=ON \ >> >> >> > -DARROW_PYTHON=ON \ >> >> >> > -DARROW_BUILD_TESTS=ON \ >> >> >> > >> >> >> -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3 >> >> \ >> >> >> > .. >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > With Regards, >> >> >> > Vibhatha Abeykoon, >> >> >> > Research Assistant, >> >> >> > Intelligent Systems Engineering, >> >> >> > Indiana University Bloomington, >> >> >> > Cell : +1-812-955-1394 >> >> >> > Web: https://www.vibhatha.org >> >> >> > >> >> >> > >> >> >> > >> >> >> > On Fri, May 1, 2020 at 11:29 AM Neal Richardson < >> >> neal.p.richardson@gmail.com> wrote: >> >> >> >> >> >> >> >> Hi Vibhatha, >> >> >> >> Did you build Arrow C++ with -DARROW_PYTHON=ON? >> >> >> >> >> >> >> >> Neal >> >> >> >> >> >> >> >> On Fri, May 1, 2020 at 8:24 AM Vibhatha Abeykoon < >> vibhatha@gmail.com> >> >> wrote: >> >> >> >>> >> >> >> >>> Hi, >> >> >> >>> >> >> >> >>> I am trying to integrate Arrow with an application that I am >> >> developing. Here I build Arrow from the source (CPP) and use the API to >> >> develop some custom functions to do a scientific calculation after data >> >> loaded with Arrow table API. On top of this, I develop a Cython API to >> >> design a python API. >> >> >> >>> >> >> >> >>> In the current stage, I have a new necessity where I need to >> >> consume Arrow Cython API for my code. >> >> >> >>> >> >> >> >>> Here It was hard to link the build libarrow.so.16 with the >> >> libarrow_python.so.16 from the installed pyarrow (separately from pip). >> >> What I realised was everything has to be built from the same source, so >> >> that I can install pyarrow from the source in my virtual environment. >> >> >> >>> >> >> >> >>> Before going through deeper things, I started by just building >> from >> >> source (CPP) and then moving towards installing pyarrow from the >> source. >> >> >> >>> >> >> >> >>> I tried to follow the guideline form here, >> >> >> >>> >> >> >> >>> https://arrow.apache.org/docs/developers/python.html, >> >> >> >>> >> >> >> >>> But when I found issues in the python build, I followed this >> source, >> >> >> >>> (but still, I used the clone from the master, not a released >> >> version) >> >> >> >>> >> >> >> >>> >> https://gist.github.com/heavyinfo/04e1326bb9bed9cecb19c2d603c8d521 >> >> >> >>> >> >> >> >>> My environmental variables are as follows, >> >> >> >>> >> >> >> >>> python3 setup.py build_ext --inplace >> >> >> >>> running build_ext >> >> >> >>> -- Running cmake for pyarrow >> >> >> >>> cmake >> >> >> -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3 >> >> -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=off >> >> -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off >> >> -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=on >> >> -DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off >> -DPYARROW_BUILD_HDFS=off >> >> -DPYARROW_USE_TENSORFLOW=off -DPYARROW_BUNDLE_ARROW_CPP=off >> >> -DPYARROW_BUNDLE_BOOST=off -DPYARROW_GENERATE_COVERAGE=off >> >> -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_PARQUET_USE_SHARED=on >> >> -DCMAKE_BUILD_TYPE=release >> /home/vibhatha/sandbox/arrow/repos/arrow/python >> >> >> >>> -- System processor: x86_64 >> >> >> >>> -- Arrow build warning level: PRODUCTION >> >> >> >>> Using ld linker >> >> >> >>> Configured for RELEASE build (set with cmake >> >> -DCMAKE_BUILD_TYPE={release,debug,...}) >> >> >> >>> -- Build Type: RELEASE >> >> >> >>> -- Build output directory: >> >> >> /home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/release >> >> >> >>> -- Arrow version: 0.18.0 (HOME: >> >> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs) >> >> >> >>> -- Arrow SO and ABI version: 18 >> >> >> >>> -- Arrow full SO version: 18.0.0 >> >> >> >>> -- Found the Arrow core shared library: >> >> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so >> >> >> >>> -- Found the Arrow core import library: >> >> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so >> >> >> >>> -- Found the Arrow core static library: >> >> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.a >> >> >> >>> CMake Error at >> >> >> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 >> >> (message): >> >> >> >>> Could NOT find ArrowPython (missing: ARROW_PYTHON_INCLUDE_DIR) >> >> (found >> >> >> >>> version "0.18.0") >> >> >> >>> Call Stack (most recent call first): >> >> >> >>> >> >> >> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 >> >> (_FPHSA_FAILURE_MESSAGE) >> >> >> >>> cmake_modules/FindArrowPython.cmake:76 >> >> (find_package_handle_standard_args) >> >> >> >>> CMakeLists.txt:210 (find_package) >> >> >> >>> >> >> >> >>> >> >> >> >>> -- Configuring incomplete, errors occurred! >> >> >> >>> See also >> >> >> "/home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeOutput.log". >> >> >> >>> error: command 'cmake' failed with exit status 1 >> >> >> >>> >> >> >> >>> Maybe I am missing some step and I am not quite sure what is the >> >> issue. >> >> >> >>> >> >> >> >>> Any pointers to solve this issue? >> >> >> >>> >> >> >> >>> With Regards, >> >> >> >>> Vibhatha >> >> >> >>> >> >> >> >>> >> >> >> > --000000000000e09fe405a49d7f67 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I will elaborate=C2=A0a couple of reasons,=C2=A0

When there are a couple of versions of Arrow,= used for different projects depending on various development choices, it i= s convenient for me to keep them pointed towards a folder of my choice.
Then refer to it and continue the work. Correct me if I am wrong,= what if I point to this folder of my choice and add it to the LD_LIBRARY_P= ATH.=C2=A0
Will this cause issues?=C2=A0

With Regards,=C2=A0
Vibhatha Abeykoon
<= /div>


On Fri, May 1, 2020 at 6:09 PM Vibhatha Abeykoon <<= a href=3D"mailto:vibhatha@gmail.com">vibhatha@gmail.com> wrote:
<= /div>
Okay, thank you for your response.=C2=A0
=
With Regards,=C2=A0
Vi= bhatha Abeykoon


On Fri, May 1, 2020 at 5:17 PM S= utou Kouhei <kou= @clear-code.com> wrote:
Hi,

> I used these settings, I still want the libs to be in a custom directo= ry,
> not in lib/

Why? We don't recommend it if you don't a CMake expert.

You can't do this if you use ARROW_HOME environment
variable.

You may able to do this by removing ARROW_HOME environment
variable and adding
PYARROW_CMAKE_OPTIONS=3D"-DCMAKE_MODULE_PATH=3D/home/vibhatha/sandbox/= arrow/repos/arrow/cpp/dist/lib/cmake/arrow"
environment variable or something.


Thanks,
--
kou

In <CABC-QTmsekZwjWG4X=3DaWrfQZgYi+qLdyEc_kvbN2njgtwJLofw@ma= il.gmail.com>
=C2=A0 "Re: PyArrow building from source issue" on Fri, 1 May 202= 0 15:05:51 -0400,
=C2=A0 Vibhatha Abeykoon <vibhatha@gmail.com> wrote:

> export ARROW_HOME=3D/home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist<= br> > export LD_LIBRARY_PATH=3D$ARROW_HOME/lib:$LD_LIBRARY_PATH
>
> export PYARROW_WITH_PARQUET=3D1
> export PYARROW_WITH_PYTHON=3D1
> export PYARROW_WITH_BUILD_TESTS=3D1
>
> cmake -DCMAKE_INSTALL_PREFIX=3D$ARROW_HOME \
>
> -DCMAKE_INSTALL_LIBDIR=3D/home/vibhatha/sandbox/arrow/repos/arrow/cpp/= arrowmylibs
> \
>=C2=A0 =C2=A0 =C2=A0 =C2=A0-DARROW_WITH_BZ2=3DOFF \
>=C2=A0 =C2=A0 =C2=A0 =C2=A0-DARROW_WITH_ZLIB=3DOFF \
>=C2=A0 =C2=A0 =C2=A0 =C2=A0-DARROW_WITH_ZSTD=3DOFF \
>=C2=A0 =C2=A0 =C2=A0 =C2=A0-DARROW_WITH_LZ4=3DOFF \
>=C2=A0 =C2=A0 =C2=A0 =C2=A0-DARROW_WITH_SNAPPY=3DOFF \
>=C2=A0 =C2=A0 =C2=A0 =C2=A0-DARROW_WITH_BROTLI=3DOFF \
>=C2=A0 =C2=A0 =C2=A0 =C2=A0-DARROW_PARQUET=3DON \
>=C2=A0 =C2=A0 =C2=A0 =C2=A0-DARROW_PYTHON=3DON \
>=C2=A0 =C2=A0 =C2=A0 =C2=A0-DARROW_BUILD_TESTS=3DON \
>
> -DPYTHON_EXECUTABLE=3D/home/vibhatha/sandbox/arrow/repos/arrow/ENVARRO= W/bin/python3
> \
>=C2=A0 =C2=A0 =C2=A0 =C2=A0..
>
>
> I used these settings, I still want the libs to be in a custom directo= ry,
> not in lib/
>
> Does it make things not work?
>
> Now I get the following error,
>
> python setup.py build_ext --inplace
> WARNING: The wheel package is not available.
> running build_ext
> -- Running cmake for pyarrow
> cmake -DPYTHON_EXECUTABLE=3D/home/vibhatha/sandbox/arrow/ENVARROW/bin/= python
>=C2=A0 -DPYARROW_BUILD_CUDA=3Doff -DPYARROW_BUILD_FLIGHT=3Doff
> -DPYARROW_BUILD_GANDIVA=3Doff -DPYARROW_BUILD_DATASET=3Doff
> -DPYARROW_BUILD_ORC=3Doff -DPYARROW_BUILD_PARQUET=3Don
> -DPYARROW_BUILD_PLASMA=3Doff -DPYARROW_BUILD_S3=3Doff -DPYARROW_BUILD_= HDFS=3Doff
> -DPYARROW_USE_TENSORFLOW=3Doff -DPYARROW_BUNDLE_ARROW_CPP=3Doff
> -DPYARROW_BUNDLE_BOOST=3Doff -DPYARROW_GENERATE_COVERAGE=3Doff
> -DPYARROW_BOOST_USE_SHARED=3Don -DPYARROW_PARQUET_USE_SHARED=3Don
> -DCMAKE_BUILD_TYPE=3Drelease /home/vibhatha/sandbox/arrow/repos/arrow/= python
> -- System processor: x86_64
> -- Arrow build warning level: PRODUCTION
> Using ld linker
> Configured for RELEASE build (set with cmake
> -DCMAKE_BUILD_TYPE=3D{release,debug,...})
> -- Build Type: RELEASE
> -- Build output directory:
> /home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_6= 4-3.8/release
> -- Arrow version: 0.18.0 (HOME:
> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist)
> -- Arrow SO and ABI version: 18
> -- Arrow full SO version: 18.0.0
> -- Found the Arrow core shared library:
> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so > -- Found the Arrow core import library:
> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so > -- Found the Arrow core static library:
> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.a > -- Found the Arrow Python by HOME:
> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist
> -- Found the Arrow Python shared library:
> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow_pyth= on.so
> -- Found the Arrow Python import library:
> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow_pyth= on.so
> -- Found the Arrow Python static library:
> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow_pyth= on.a
> CMake Error at
> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmak= e:146
> (message):
>=C2=A0 =C2=A0Could NOT find Parquet (missing: PARQUET_LIB_DIR) (found v= ersion "1.5.1")
> Call Stack (most recent call first):
>
> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmak= e:393
> (_FPHSA_FAILURE_MESSAGE)
>=C2=A0 =C2=A0cmake_modules/FindParquet.cmake:116 (find_package_handle_s= tandard_args)
>=C2=A0 =C2=A0CMakeLists.txt:426 (find_package)
>
> Does Parquet need to be installed separately?
>
> With Regards,
> Vibhatha Abeykoon,
> Research Assistant,
> Intelligent Systems Engineering,
> Indiana University Bloomington,
> Cell : +1-812-955-1394
> Web: https://www.vibhatha.org
> <https://www.linkedin.com/in/vibhathaabeykoon/<= /a>>
>
>
> On Fri, May 1, 2020 at 1:55 PM Wes McKinney <wesmckinn@gmail.com> wrote:
>
>> Try this instead
>>
>> export ARROW_HOME=3D/home/vibhatha/sandbox/arrow/repos/arrow/cpp/d= ist
>> export LD_LIBRARY_PATH=3D$ARROW_HOME/lib:$LD_LIBRARY_PATH
>>
>> Make sure to use
>>
>> -DCMAKE_INSTALL_PREFIX=3D$ARROW_HOME -DCMAKE_INSTALL_LIBDIR=3Dlib<= br> >>
>> I renamed from "arrowmylibs" to "dist" so that= it's less confusing
>> what that directory represents (it doesn't contains the libs >> directory, but rather the directories include/, lib/, etc.)
>>
>>
>> On Fri, May 1, 2020 at 12:51 PM Vibhatha Abeykoon <vibhatha@gmail.com>
>> wrote:
>> >
>> > I didn't clearly mention the config I used,
>> >
>> > export
>> ARROW_HOME=3D/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmyl= ibs
>> > export LD_LIBRARY_PATH=3D$ARROW_HOME:$LD_LIBRARY_PATH
>> >
>> > Before starting I added these ENV vars.
>> >
>> > My objective is to get all the libs inside my build setting. = I use arrow
>> as one of the core libraries in my project.
>> > I have to build it in the following way,
>> >
>> > 1. Arrow clone from git
>> > 2. Arrow CPP built
>> > 3. Using arrow CPP I develop custom functions upon loaded dat= a
>> > 4. Integrate Cython APIs for these custom functions
>> > 5. Use Pyarrow Cython to provide more functionality from Pyth= on end to
>> Cython to my CPP lib which uses Arrow.
>> >
>> > This is the build that I am trying to formulate. So I have to= keep those
>> libs there, and the idea is to build python from
>> > the same cloned source. The challenge was to keep all the sha= red libs
>> from arrow, my-cython libs to point in the right
>> > direction. Is this a clear description?
>> >
>> > With Regards,
>> > Vibhatha Abeykoon,
>> > Research Assistant,
>> > Intelligent Systems Engineering,
>> > Indiana University Bloomington,
>> > Cell : +1-812-955-1394
>> > Web: https://www.vibhatha.org
>> >
>> >
>> >
>> > On Fri, May 1, 2020 at 1:28 PM Wes McKinney <wesmckinn@gmail.com> wro= te:
>> >>
>> >> This part doesn't look correct
>> >>
>> >> -DCMAKE_INSTALL_PREFIX=3D$ARROW_HOME \
>> >>
>>=C2=A0 -DCMAKE_INSTALL_LIBDIR=3D/home/vibhatha/sandbox/arrow/repos/= arrow/cpp/arrowmylibs
>> >>
>> >> The usual incantation is
>> >>
>> >> -DCMAKE_INSTALL_PREFIX=3D$ARROW_HOME \
>> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0-DCMAKE_INSTALL_LIBDIR=3Dlib >> >>
>> >> The reason we pass -DCMAKE_INSTALL_LIBDIR=3Dlib is that s= ome systems
>> >> will install libraries in lib32 or lib64 instead of just = lib
>> >>
>> >>
>> >> On Fri, May 1, 2020 at 10:38 AM Vibhatha Abeykoon <vibhatha@gmail.com= >
>> wrote:
>> >> >
>> >> > Hi Neal,
>> >> >
>> >> > Yes, I added the flag. But I installed my libs not t= o /usr/lib, but
>> to a different folder.
>> >> >
>> >> > This is the way I built arrow C++,
>> >> >
>> >> > cmake -DCMAKE_INSTALL_PREFIX=3D$ARROW_HOME \
>> >> >
>>=C2=A0 -DCMAKE_INSTALL_LIBDIR=3D/home/vibhatha/sandbox/arrow/repos/= arrow/cpp/arrowmylibs
>> \
>> >> >=C2=A0 =C2=A0 =C2=A0 =C2=A0-DARROW_WITH_BZ2=3DOFF \ >> >> >=C2=A0 =C2=A0 =C2=A0 =C2=A0-DARROW_WITH_ZLIB=3DOFF \<= br> >> >> >=C2=A0 =C2=A0 =C2=A0 =C2=A0-DARROW_WITH_ZSTD=3DOFF \<= br> >> >> >=C2=A0 =C2=A0 =C2=A0 =C2=A0-DARROW_WITH_LZ4=3DOFF \ >> >> >=C2=A0 =C2=A0 =C2=A0 =C2=A0-DARROW_WITH_SNAPPY=3DOFF = \
>> >> >=C2=A0 =C2=A0 =C2=A0 =C2=A0-DARROW_WITH_BROTLI=3DOFF = \
>> >> >=C2=A0 =C2=A0 =C2=A0 =C2=A0-DARROW_PARQUET=3DON \
>> >> >=C2=A0 =C2=A0 =C2=A0 =C2=A0-DARROW_PYTHON=3DON \
>> >> >=C2=A0 =C2=A0 =C2=A0 =C2=A0-DARROW_BUILD_TESTS=3DON \=
>> >> >
>>=C2=A0 -DPYTHON_EXECUTABLE=3D/home/vibhatha/sandbox/arrow/repos/arr= ow/ENVARROW/bin/python3
>> \
>> >> >=C2=A0 =C2=A0 =C2=A0 =C2=A0..
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > With Regards,
>> >> > Vibhatha Abeykoon,
>> >> > Research Assistant,
>> >> > Intelligent Systems Engineering,
>> >> > Indiana University Bloomington,
>> >> > Cell : +1-812-955-1394
>> >> > Web: https://www.vibhatha.org
>> >> >
>> >> >
>> >> >
>> >> > On Fri, May 1, 2020 at 11:29 AM Neal Richardson <=
>> n= eal.p.richardson@gmail.com> wrote:
>> >> >>
>> >> >> Hi Vibhatha,
>> >> >> Did you build Arrow C++ with -DARROW_PYTHON=3DON= ?
>> >> >>
>> >> >> Neal
>> >> >>
>> >> >> On Fri, May 1, 2020 at 8:24 AM Vibhatha Abeykoon= <vibhatha@gmail= .com>
>> wrote:
>> >> >>>
>> >> >>> Hi,
>> >> >>>
>> >> >>> I am trying to integrate Arrow with an appli= cation that I am
>> developing. Here I build Arrow from the source (CPP) and use the A= PI to
>> develop some custom functions to do a scientific calculation after= data
>> loaded with Arrow table API. On top of this, I develop a Cython AP= I to
>> design a python API.
>> >> >>>
>> >> >>> In the current stage, I have a new necessity= where I need to
>> consume Arrow Cython API for my code.
>> >> >>>
>> >> >>> Here It was hard to link the build libarrow.= so.16 with the
>> libarrow_python.so.16 from the installed pyarrow (separately from = pip).
>> What I realised was everything has to be built from the same sourc= e, so
>> that I can install pyarrow from the source in my virtual environme= nt.
>> >> >>>
>> >> >>> Before going through deeper things, I starte= d by just building from
>> source (CPP) and then moving towards installing pyarrow from the s= ource.
>> >> >>>
>> >> >>> I tried to follow the guideline form here, >> >> >>>
>> >> >>> https://arrow.apa= che.org/docs/developers/python.html,
>> >> >>>
>> >> >>> But when I found issues in the python build,= I followed this source,
>> >> >>> (but still, I used the clone from the master= , not a released
>> version)
>> >> >>>
>> >> >>> htt= ps://gist.github.com/heavyinfo/04e1326bb9bed9cecb19c2d603c8d521
>> >> >>>
>> >> >>> My environmental variables are as follows, >> >> >>>
>> >> >>> python3 setup.py build_ext --inplace
>> >> >>> running build_ext
>> >> >>> -- Running cmake for pyarrow
>> >> >>> cmake
>> -DPYTHON_EXECUTABLE=3D/home/vibhatha/sandbox/arrow/repos/arrow/ENV= ARROW/bin/python3
>> -DPYARROW_BUILD_CUDA=3Doff -DPYARROW_BUILD_FLIGHT=3Doff
>> -DPYARROW_BUILD_GANDIVA=3Doff -DPYARROW_BUILD_DATASET=3Doff
>> -DPYARROW_BUILD_ORC=3Doff -DPYARROW_BUILD_PARQUET=3Don
>> -DPYARROW_BUILD_PLASMA=3Doff -DPYARROW_BUILD_S3=3Doff -DPYARROW_BU= ILD_HDFS=3Doff
>> -DPYARROW_USE_TENSORFLOW=3Doff -DPYARROW_BUNDLE_ARROW_CPP=3Doff >> -DPYARROW_BUNDLE_BOOST=3Doff -DPYARROW_GENERATE_COVERAGE=3Doff
>> -DPYARROW_BOOST_USE_SHARED=3Don -DPYARROW_PARQUET_USE_SHARED=3Don<= br> >> -DCMAKE_BUILD_TYPE=3Drelease /home/vibhatha/sandbox/arrow/repos/ar= row/python
>> >> >>> -- System processor: x86_64
>> >> >>> -- Arrow build warning level: PRODUCTION
>> >> >>> Using ld linker
>> >> >>> Configured for RELEASE build (set with cmake=
>> -DCMAKE_BUILD_TYPE=3D{release,debug,...})
>> >> >>> -- Build Type: RELEASE
>> >> >>> -- Build output directory:
>> /home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x= 86_64-3.8/release
>> >> >>> -- Arrow version: 0.18.0 (HOME:
>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs)
>> >> >>> -- Arrow SO and ABI version: 18
>> >> >>> -- Arrow full SO version: 18.0.0
>> >> >>> -- Found the Arrow core shared library:
>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.= so
>> >> >>> -- Found the Arrow core import library:
>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.= so
>> >> >>> -- Found the Arrow core static library:
>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.= a
>> >> >>> CMake Error at
>> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.= cmake:146
>> (message):
>> >> >>>=C2=A0 =C2=A0Could NOT find ArrowPython (miss= ing: ARROW_PYTHON_INCLUDE_DIR)
>> (found
>> >> >>>=C2=A0 =C2=A0version "0.18.0")
>> >> >>> Call Stack (most recent call first):
>> >> >>>
>>=C2=A0 /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandar= dArgs.cmake:393
>> (_FPHSA_FAILURE_MESSAGE)
>> >> >>>=C2=A0 =C2=A0cmake_modules/FindArrowPython.cm= ake:76
>> (find_package_handle_standard_args)
>> >> >>>=C2=A0 =C2=A0CMakeLists.txt:210 (find_package= )
>> >> >>>
>> >> >>>
>> >> >>> -- Configuring incomplete, errors occurred!<= br> >> >> >>> See also
>> "/home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.l= inux-x86_64-3.8/CMakeFiles/CMakeOutput.log".
>> >> >>> error: command 'cmake' failed with e= xit status 1
>> >> >>>
>> >> >>> Maybe I am missing some step and I am not qu= ite sure what is the
>> issue.
>> >> >>>
>> >> >>> Any pointers to solve this issue?
>> >> >>>
>> >> >>> With Regards,
>> >> >>> Vibhatha
>> >> >>>
>> >> >>>
>>
--000000000000e09fe405a49d7f67--