arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Mior <michael.m...@gmail.com>
Subject Re: Arrow/Gandiva dependency management in Java
Date Tue, 05 Jan 2021 18:41:42 GMT
You're correct that AppVeyor is Windows. I think it's fine if Windows
isn't supported for the time being. I assume you mean a JIRA issue as
I'm not able to open issues on the Arrow GH. Link to the JIRA issue
below. I'll be adding more details. Unfortunately still waiting on
Travis.

https://issues.apache.org/jira/browse/ARROW-11135

--
Michael Mior
michael.mior@gmail.com

Le mar. 5 janv. 2021 à 13:09, Jacques Nadeau <jacques@apache.org> a écrit :
>
> If I recall correctly, our artifact doesn't support windows (just mac and linux). If
AppVeyor is for windows I would assume that would fail.
>
> Let's see what happens with Travis. I'd expect it to succeed.
>
> I suggest you open a Arrow GH issue to track this problem and we continue to record the
result of items there as well.
>
> Can you also confirm the difference of the set of native libraries in the Dremio jar
versus the Apache one?
>
>
>
>
> On Tue, Jan 5, 2021 at 9:49 AM Michael Mior <michael.mior@gmail.com> wrote:
>>
>> Thanks Jaques! Using the Dremio artifacts do seem to work on my macOS
>> development machine but failed with the same error on AppVeyor. (Still
>> waiting on Travis.) If there's anything I can do to help with
>> debugging, please let me know! You can see a (somewhat messy) PR in
>> progress at the link below. The last commit on the branch switches to
>> the Dremio repository if you want to try both and see if you can
>> reproduce (./gradlew :arrow:test should succeed if things are working
>> correctly).
>>
>> https://github.com/apache/calcite/pull/2133
>>
>> --
>> Michael Mior
>> michael.mior@gmail.com
>>
>> Le mar. 5 janv. 2021 à 12:08, Jacques Nadeau <jacques@apache.org> a écrit
:
>> >
>> > If you're up for troubleshooting, do you want to try the near master jar we
use inside of Dremio? I think we're the primary users of the Gandiva Java bindings and I wonder
if the Apache build has an issue with artifact packaging(?). The jar we use in Dremio is used
at hundreds of customers so it should work without extra items. You can find the reference
in the pom here:
>> >
>> > https://github.com/dremio/dremio-oss/blob/master/pom.xml#L41
>> >
>> > I think you'll need to add our public repo to get the jar. I think it is this
repo:
>> > https://github.com/dremio/dremio-oss/blob/master/pom.xml#L3243
>> >
>> > I'm only suggesting this to see if there is some kind of disconnect in keeping
the Apache artifact complete. If our jar works for you we'll need to figure out what is different
between it and the Apache one. (For reference, in general our jars are slightly off master
to pull forward or backward certain bugs, etc.)
>> >
>> >
>> >
>> > On Tue, Jan 5, 2021 at 8:55 AM Michael Mior <mmior@apache.org> wrote:
>> >>
>> >> Yes, I'm using the latest released version (2.0.0).
>> >> --
>> >> Michael Mior
>> >> mmior@apache.org
>> >>
>> >> Le mar. 5 janv. 2021 à 11:45, Jacques Nadeau <jacques@apache.org>
a écrit :
>> >> >
>> >> > Hey Michael, are you using the released versions of these Arrow artifacts?
I recall there being complexity to building the multi-native-binary jar file...
>> >> >
>> >> > On Tue, Jan 5, 2021 at 8:11 AM Michael Mior <mmior@apache.org>
wrote:
>> >> >>
>> >> >> I am working on integrating Arrow and Gandiva with the Apache Calcite
>> >> >> query processing framework so Calcite is able to process SQL queries
>> >> >> using Arrow data sources. The basics are working well but I'm having
>> >> >> trouble getting a reproducible development environment set up.
>> >> >>
>> >> >> For example, on a Linux Travis CI right now, I get the following
error:
>> >> >>
>> >> >> java.lang.RuntimeException: libgandiva_jni.so was not found inside
JAR.
>> >> >>
>> >> >> I get something similar with AppVeyor on Windows:
>> >> >>
>> >> >> java.lang.RuntimeException: gandiva_jni.dll was not found inside
JAR.
>> >> >>
>> >> >> Starting with a fresh clone of the repository, on my macOS machine
I
>> >> >> use for development, I get the following which seems to be related
to
>> >> >> a protobuf version mismatch:
>> >> >>
>> >> >> java.lang.UnsatisfiedLinkError:
>> >> >> /private/var/folders/0h/n0jrz4qn5x37gj25sxx4z2wm0000gp/T/libgandiva_jni.dylib43feded5-f265-4a6d-b1f7-2aa56e94f392:
>> >> >> dlopen(/private/var/folders/0h/n0jrz4qn5x37gj25sxx4z2wm0000gp/T/libgandiva_jni.dylib43feded5-f265-4a6d-b1f7-2aa56e94f392,
>> >> >> 1): Library not loaded:
>> >> >> /usr/local/opt/protobuf/lib/libprotobuf.24.dylib
>> >> >>       Referenced from:
>> >> >> /private/var/folders/0h/n0jrz4qn5x37gj25sxx4z2wm0000gp/T/libgandiva_jni.dylib43feded5-f265-4a6d-b1f7-2aa56e94f392
>> >> >>
>> >> >> I have dependencies declared on arrow-memory-netty, arrow-vector,
and
>> >> >> arrow-gandiva but this doesn't seem to be sufficient. Any ideas
what
>> >> >> I'm missing here?
>> >> >>
>> >> >> --
>> >> >> Michael Mior
>> >> >> mmior@apache.org

Mime
View raw message