From user-return-408-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Fri May 1 15:24:03 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id C103B180676 for ; Fri, 1 May 2020 17:24:02 +0200 (CEST) Received: (qmail 51197 invoked by uid 500); 1 May 2020 15:24:02 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 51176 invoked by uid 99); 1 May 2020 15:24:02 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2020 15:24:02 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 8CC9718125B for ; Fri, 1 May 2020 15:24:01 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.01 X-Spam-Level: X-Spam-Status: No, score=0.01 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_KAM_HTML_FONT_INVALID=0.01, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id FFrO_3xXDtAK for ; Fri, 1 May 2020 15:23:59 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.219.174; helo=mail-yb1-f174.google.com; envelope-from=vibhatha@gmail.com; receiver= Received: from mail-yb1-f174.google.com (mail-yb1-f174.google.com [209.85.219.174]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 44F17BB808 for ; Fri, 1 May 2020 15:23:59 +0000 (UTC) Received: by mail-yb1-f174.google.com with SMTP id b17so5095862ybq.13 for ; Fri, 01 May 2020 08:23:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=nI0ffh/iivNcmVxlbTQY8WRjYka04HiFz9ySkx/oy80=; b=qn1NGqvyHxoFRfPfJ79ddF+GkuS3DpUC6KLBhCN+peMsrwMiTDhNvvy0dqSpIE4pE7 1mCpIzNcig430221h/LjwV2Z2YKAejCt6mZdyKHEjsRDLiNIKua/RV8yW6H0+DtZUAeB WMMOzEXKix0y21zyntQmiYwAG0ep1JtamBT7xQBT77CYSjrGV1DfCK9EU+pr4Wu2G6cj y+gM+c+mX79hWiKvwkS7e9rQILPyrfuzohJW1fOBkjRl4Ju0ZrvV7oZip/Q3nrxr4rsG x3DsqoS7+5cy/r+DQ1Vtd90uakPfrfy86ATMQj45xIZLFZrh46R4CM/hrGJA9V05C7Fa mR9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=nI0ffh/iivNcmVxlbTQY8WRjYka04HiFz9ySkx/oy80=; b=KVcIWpUTyWdUxEuNdW3FRbo46arjg+g8PXFngd6SmD7EEUcs0f7YEZITPL1V3iUEqc b8CJyB9kE33dxE7LaDbJsy/5YxrnCbKVHDfFnZNmbjq4RE7scYJ019EUJYUg9WExZJKG GhBlELiIqRlC+TKDM0FIZ4H1Zs+H3JjvjCXjEe4Z8CzaQ2VxssgJ52SHQqBaA3c3nSVN b48xIuhYt6fCOEVtmAFsYvOitbCJNTl6xYSplKFRSYT0fqd5qjtp0G41EFYI4Bt0MLJR h9lUbNAgJcZ3KsPf7jdy7BgVoj2m8nAM3Oi3BI91WufWQpZPqVa93EP2AmhCh33Svjn1 +RVw== X-Gm-Message-State: AGi0PubhBFSrtdOnpTvKblwB6KQ6T6OdI3qJIMwYQ8CRUoWVlYeYIuO/ Miy2pCb1+J2vaVfWoISwI5mH7BTLZmNsEILjbQXy8UG6W70= X-Google-Smtp-Source: APiQypLlel44r34kASAM2Ke+4JUzy4xrHFTmA0BtIbLcI7gTtNQJsCjwgsdTywHTY2OPehGR5ZCnQK6TpmM1Fau1fjY= X-Received: by 2002:a25:dd03:: with SMTP id u3mr6909152ybg.208.1588346631388; Fri, 01 May 2020 08:23:51 -0700 (PDT) MIME-Version: 1.0 From: Vibhatha Abeykoon Date: Fri, 1 May 2020 11:23:40 -0400 Message-ID: Subject: PyArrow building from source issue To: user@arrow.apache.org Content-Type: multipart/alternative; boundary="000000000000be860505a497c477" --000000000000be860505a497c477 Content-Type: text/plain; charset="UTF-8" Hi, I am trying to integrate Arrow with an application that I am developing. Here I build Arrow from the source (CPP) and use the API to develop some custom functions to do a scientific calculation after data loaded with Arrow table API. On top of this, I develop a Cython API to design a python API. In the current stage, I have a new necessity where I need to consume Arrow Cython API for my code. Here It was hard to link the build libarrow.so.16 with the libarrow_python.so.16 from the installed pyarrow (separately from pip). What I realised was everything has to be built from the same source, so that I can install pyarrow from the source in my virtual environment. Before going through deeper things, I started by just building from source (CPP) and then moving towards installing pyarrow from the source. I tried to follow the guideline form here, https://arrow.apache.org/docs/developers/python.html, But when I found issues in the python build, I followed this source, (but still, I used the clone from the master, not a released version) https://gist.github.com/heavyinfo/04e1326bb9bed9cecb19c2d603c8d521 My environmental variables are as follows, python3 setup.py build_ext --inplace running build_ext -- Running cmake for pyarrow cmake -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3 -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=off -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=on -DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off -DPYARROW_BUILD_HDFS=off -DPYARROW_USE_TENSORFLOW=off -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_BOOST=off -DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_PARQUET_USE_SHARED=on -DCMAKE_BUILD_TYPE=release /home/vibhatha/sandbox/arrow/repos/arrow/python -- System processor: x86_64 -- Arrow build warning level: PRODUCTION Using ld linker Configured for RELEASE build (set with cmake -DCMAKE_BUILD_TYPE={release,debug,...}) -- Build Type: RELEASE -- Build output directory: /home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/release -- Arrow version: 0.18.0 (HOME: /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs) -- Arrow SO and ABI version: 18 -- Arrow full SO version: 18.0.0 -- Found the Arrow core shared library: /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so -- Found the Arrow core import library: /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so -- Found the Arrow core static library: /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.a CMake Error at /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message): Could NOT find ArrowPython (missing: ARROW_PYTHON_INCLUDE_DIR) (found version "0.18.0") Call Stack (most recent call first): /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE) cmake_modules/FindArrowPython.cmake:76 (find_package_handle_standard_args) CMakeLists.txt:210 (find_package) -- Configuring incomplete, errors occurred! See also "/home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeOutput.log". error: command 'cmake' failed with exit status 1 Maybe I am missing some step and I am not quite sure what is the issue. Any pointers to solve this issue? With Regards, Vibhatha --000000000000be860505a497c477 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,=C2=A0

<= div class=3D"gmail_default">I am trying to integrate Arrow with an application that I am developing= . Here I build Arrow from the source (CPP) and use the API to develop some = custom functions to do a scientific calculation after data loaded with Arro= w table API. On top of this, I develop a Cython API to design a python API.=

= In the current stage, I have a new necessity where I need to consume Arrow = Cython API for my code.

Here It was hard to= link the build libarrow.so.16 with the libarrow_python.so.16 from the inst= alled pyarrow (separately from pip). What I realised was everything has to = be built from the same source, so that I can install pyarrow from the sourc= e in my virtual environment.

Before going t= hrough deeper things, I started by just building from source (CPP) and then= moving towards installing pyarrow from the source.

I tried to follow the guideline form here,

https://arro= w.apache.org/docs/developers/python.html,

But when I found issues in the python build, I followed this source,
(but still, I used the clone from the maste= r, not a released version)

https://gist.github.com= /heavyinfo/04e1326bb9bed9cecb19c2d603c8d521

My environmental variables are as follows,

python3 setup.py build_ext --inplace
running build_ext
-- Running cmake for pyarrow
cmake -DPYTHON_EXECUTABLE=3D/home/vibhatha/sandbox/arrow/repos/arrow/ENVARR=
OW/bin/python3  -DPYARROW_BUILD_CUDA=3Doff -DPYARROW_BUILD_FLIGHT=3Doff -DP=
YARROW_BUILD_GANDIVA=3Doff -DPYARROW_BUILD_DATASET=3Doff -DPYARROW_BUILD_OR=
C=3Doff -DPYARROW_BUILD_PARQUET=3Don -DPYARROW_BUILD_PLASMA=3Doff -DPYARROW=
_BUILD_S3=3Doff -DPYARROW_BUILD_HDFS=3Doff -DPYARROW_USE_TENSORFLOW=3Doff -=
DPYARROW_BUNDLE_ARROW_CPP=3Doff -DPYARROW_BUNDLE_BOOST=3Doff -DPYARROW_GENE=
RATE_COVERAGE=3Doff -DPYARROW_BOOST_USE_SHARED=3Don -DPYARROW_PARQUET_USE_S=
HARED=3Don -DCMAKE_BUILD_TYPE=3Drelease /home/vibhatha/sandbox/arrow/repos/=
arrow/python
-- System processor: x86_64
-- Arrow build warning level: PRODUCTION
Using ld linker
Configured for RELEASE build (set with cmake -DCMAKE_BUILD_TYPE=3D{release,debug,...=
})
-- Build Type: RELEASE
-- Build output directory: /home/vibhatha/sandbox/arrow/repos/arrow/python/=
build/temp.linux-x86_64-3.8/release
-- Arrow version: 0.18.0 (HOME: /home/vibhatha/sandbox/arrow/repos/arrow/cp=
p/arrowmylibs)
-- Arrow SO and ABI version: 18
-- Arrow full SO version: 18.0.0
-- Found the Arrow core shared library: /home/vibhatha/sandbox/arrow/repos/=
arrow/cpp/arrowmylibs/libarrow.so
-- Found the Arrow core import library: /home/vibhatha/sandbox/arrow/repos/=
arrow/cpp/arrowmylibs/libarrow.so
-- Found the Arrow core static library: /home/vibhatha/sandbox/arrow/repos/=
arrow/cpp/arrowmylibs/libarrow.a
CMake Error at /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandar=
dArgs.cmake:146 (message):
  Could NOT find ArrowPython (missing: ARROW_PYTHON_INCLUDE_DIR) (found
  version "0.18.0")
Call Stack (most recent call first):
  /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:3=
93 (_FPHSA_FAILURE_MESSAGE)
  cmake_modules/FindArrowPython.cmake:76 (find_package_handle_standard_args=
)
  CMakeLists.txt:210 (find_package)


-- Configuring incomplete, errors occurred!
See also "/home/vibhatha/sandbox/arrow/repo=
s/arrow/python/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeOutput.log".
error: command 'cmake' failed with exit status 1
<= div dir=3D"ltr">
Maybe I am missing some= step and I am not quite sure what is the issue.=C2=A0
<= div>
Any pointers to solve = this issue?=C2=A0

With Regar= ds,=C2=A0
Vibhatha


--000000000000be860505a497c477--