From user-return-902-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Tue Jan 5 17:07:42 2021 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-ec2-va.apache.org (mxout1-ec2-va.apache.org [3.227.148.255]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 2FE12180634 for ; Tue, 5 Jan 2021 18:07:42 +0100 (CET) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-ec2-va.apache.org (ASF Mail Server at mxout1-ec2-va.apache.org) with SMTP id F0D0643BFB for ; Tue, 5 Jan 2021 17:07:40 +0000 (UTC) Received: (qmail 60878 invoked by uid 500); 5 Jan 2021 17:07:40 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 60867 invoked by uid 99); 5 Jan 2021 17:07:40 -0000 Received: from mailrelay1-he-de.apache.org (HELO mailrelay1-he-de.apache.org) (116.203.21.61) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2021 17:07:40 +0000 Received: from mail-oi1-f176.google.com (mail-oi1-f176.google.com [209.85.167.176]) by mailrelay1-he-de.apache.org (ASF Mail Server at mailrelay1-he-de.apache.org) with ESMTPSA id 79CA73E83E for ; Tue, 5 Jan 2021 17:07:39 +0000 (UTC) Received: by mail-oi1-f176.google.com with SMTP id f132so218160oib.12 for ; Tue, 05 Jan 2021 09:07:39 -0800 (PST) X-Gm-Message-State: AOAM532mV75X1HMECtwodr1U7jPKZnnuZG0OCVz1bovhYsxxC8+pvgen uO0HBByQ5F8VeHZ4pmEM2vWAlOXfIknbil2PnF8= X-Google-Smtp-Source: ABdhPJzKf48T7lLQjywF2rL/rbgXaOdBYYpojwihi/YAbUf2DC00wzW6zFe+Fq/FQPuJ/D5NXc1tc0iJ7n16lSn6Z3w= X-Received: by 2002:aca:3151:: with SMTP id x78mr365685oix.39.1609866458236; Tue, 05 Jan 2021 09:07:38 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Jacques Nadeau Date: Tue, 5 Jan 2021 09:07:27 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Arrow/Gandiva dependency management in Java To: user@arrow.apache.org Content-Type: multipart/alternative; boundary="000000000000610de705b82a3ec1" --000000000000610de705b82a3ec1 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable If you're up for troubleshooting, do you want to try the near master jar we use inside of Dremio? I think we're the primary users of the Gandiva Java bindings and I wonder if the Apache build has an issue with artifact packaging(?). The jar we use in Dremio is used at hundreds of customers so it should work without extra items. You can find the reference in the pom here: https://github.com/dremio/dremio-oss/blob/master/pom.xml#L41 I think you'll need to add our public repo to get the jar. I think it is this repo: https://github.com/dremio/dremio-oss/blob/master/pom.xml#L3243 I'm only suggesting this to see if there is some kind of disconnect in keeping the Apache artifact complete. If our jar works for you we'll need to figure out what is different between it and the Apache one. (For reference, in general our jars are slightly off master to pull forward or backward certain bugs, etc.) On Tue, Jan 5, 2021 at 8:55 AM Michael Mior wrote: > Yes, I'm using the latest released version (2.0.0). > -- > Michael Mior > mmior@apache.org > > Le mar. 5 janv. 2021 =C3=A0 11:45, Jacques Nadeau a = =C3=A9crit > : > > > > Hey Michael, are you using the released versions of these Arrow > artifacts? I recall there being complexity to building the > multi-native-binary jar file... > > > > On Tue, Jan 5, 2021 at 8:11 AM Michael Mior wrote: > >> > >> I am working on integrating Arrow and Gandiva with the Apache Calcite > >> query processing framework so Calcite is able to process SQL queries > >> using Arrow data sources. The basics are working well but I'm having > >> trouble getting a reproducible development environment set up. > >> > >> For example, on a Linux Travis CI right now, I get the following error= : > >> > >> java.lang.RuntimeException: libgandiva_jni.so was not found inside JAR= . > >> > >> I get something similar with AppVeyor on Windows: > >> > >> java.lang.RuntimeException: gandiva_jni.dll was not found inside JAR. > >> > >> Starting with a fresh clone of the repository, on my macOS machine I > >> use for development, I get the following which seems to be related to > >> a protobuf version mismatch: > >> > >> java.lang.UnsatisfiedLinkError: > >> > /private/var/folders/0h/n0jrz4qn5x37gj25sxx4z2wm0000gp/T/libgandiva_jni.d= ylib43feded5-f265-4a6d-b1f7-2aa56e94f392: > >> > dlopen(/private/var/folders/0h/n0jrz4qn5x37gj25sxx4z2wm0000gp/T/libgandiv= a_jni.dylib43feded5-f265-4a6d-b1f7-2aa56e94f392, > >> 1): Library not loaded: > >> /usr/local/opt/protobuf/lib/libprotobuf.24.dylib > >> Referenced from: > >> > /private/var/folders/0h/n0jrz4qn5x37gj25sxx4z2wm0000gp/T/libgandiva_jni.d= ylib43feded5-f265-4a6d-b1f7-2aa56e94f392 > >> > >> I have dependencies declared on arrow-memory-netty, arrow-vector, and > >> arrow-gandiva but this doesn't seem to be sufficient. Any ideas what > >> I'm missing here? > >> > >> -- > >> Michael Mior > >> mmior@apache.org > --000000000000610de705b82a3ec1 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
If you're up for troubleshooting, do you want to try t= he near master jar we use inside of Dremio? I think we're the primary u= sers of the Gandiva Java bindings and I wonder if the Apache build has an i= ssue with artifact packaging(?). The jar we use in Dremio is used at hundre= ds of customers so it should work without extra items. You can find the ref= erence in the pom here:


I think you'll = need to add our public repo to get the jar. I think it is this repo:
<= div>https://github.com/dremio/dremio-oss/blob/master/pom.xml#L3243
<= /div>

I'm only suggesting this to see if there = is some kind of disconnect in keeping the Apache artifact complete. If our = jar works for you we'll need to figure out what is different between it= and the Apache one. (For reference, in general our jars are slightly off m= aster to pull forward or backward certain bugs, etc.)



On Tue, Jan 5, 2021 at 8:55 AM Michael Mior <mmior@apache.org> wrote:
<= blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-l= eft:1px solid rgb(204,204,204);padding-left:1ex">Yes, I'm using the lat= est released version (2.0.0).
--
Michael Mior
mmior@apache.org<= br>
Le mar. 5 janv. 2021 =C3=A0 11:45, Jacques Nadeau <jacques@apache.org> a =C3=A9crit = :
>
> Hey Michael, are you using the released versions of these Arrow artifa= cts? I recall there being complexity to building the multi-native-binary ja= r file...
>
> On Tue, Jan 5, 2021 at 8:11 AM Michael Mior <mmior@apache.org> wrote:
>>
>> I am working on integrating Arrow and Gandiva with the Apache Calc= ite
>> query processing framework so Calcite is able to process SQL queri= es
>> using Arrow data sources. The basics are working well but I'm = having
>> trouble getting a reproducible development environment set up.
>>
>> For example, on a Linux Travis CI right now, I get the following e= rror:
>>
>> java.lang.RuntimeException: libgandiva_jni.so was not found inside= JAR.
>>
>> I get something similar with AppVeyor on Windows:
>>
>> java.lang.RuntimeException: gandiva_jni.dll was not found inside J= AR.
>>
>> Starting with a fresh clone of the repository, on my macOS machine= I
>> use for development, I get the following which seems to be related= to
>> a protobuf version mismatch:
>>
>> java.lang.UnsatisfiedLinkError:
>> /private/var/folders/0h/n0jrz4qn5x37gj25sxx4z2wm0000gp/T/libgandiv= a_jni.dylib43feded5-f265-4a6d-b1f7-2aa56e94f392:
>> dlopen(/private/var/folders/0h/n0jrz4qn5x37gj25sxx4z2wm0000gp/T/li= bgandiva_jni.dylib43feded5-f265-4a6d-b1f7-2aa56e94f392,
>> 1): Library not loaded:
>> /usr/local/opt/protobuf/lib/libprotobuf.24.dylib
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0Referenced from:
>> /private/var/folders/0h/n0jrz4qn5x37gj25sxx4z2wm0000gp/T/libgandiv= a_jni.dylib43feded5-f265-4a6d-b1f7-2aa56e94f392
>>
>> I have dependencies declared on arrow-memory-netty, arrow-vector, = and
>> arrow-gandiva but this doesn't seem to be sufficient. Any idea= s what
>> I'm missing here?
>>
>> --
>> Michael Mior
>> mmior@apache= .org
--000000000000610de705b82a3ec1--