arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Mior <michael.m...@gmail.com>
Subject Re: Arrow/Gandiva dependency management in Java
Date Tue, 05 Jan 2021 17:49:18 GMT
Thanks Jaques! Using the Dremio artifacts do seem to work on my macOS
development machine but failed with the same error on AppVeyor. (Still
waiting on Travis.) If there's anything I can do to help with
debugging, please let me know! You can see a (somewhat messy) PR in
progress at the link below. The last commit on the branch switches to
the Dremio repository if you want to try both and see if you can
reproduce (./gradlew :arrow:test should succeed if things are working
correctly).

https://github.com/apache/calcite/pull/2133

--
Michael Mior
michael.mior@gmail.com

Le mar. 5 janv. 2021 à 12:08, Jacques Nadeau <jacques@apache.org> a écrit :
>
> If you're up for troubleshooting, do you want to try the near master jar we use inside
of Dremio? I think we're the primary users of the Gandiva Java bindings and I wonder if the
Apache build has an issue with artifact packaging(?). The jar we use in Dremio is used at
hundreds of customers so it should work without extra items. You can find the reference in
the pom here:
>
> https://github.com/dremio/dremio-oss/blob/master/pom.xml#L41
>
> I think you'll need to add our public repo to get the jar. I think it is this repo:
> https://github.com/dremio/dremio-oss/blob/master/pom.xml#L3243
>
> I'm only suggesting this to see if there is some kind of disconnect in keeping the Apache
artifact complete. If our jar works for you we'll need to figure out what is different between
it and the Apache one. (For reference, in general our jars are slightly off master to pull
forward or backward certain bugs, etc.)
>
>
>
> On Tue, Jan 5, 2021 at 8:55 AM Michael Mior <mmior@apache.org> wrote:
>>
>> Yes, I'm using the latest released version (2.0.0).
>> --
>> Michael Mior
>> mmior@apache.org
>>
>> Le mar. 5 janv. 2021 à 11:45, Jacques Nadeau <jacques@apache.org> a écrit
:
>> >
>> > Hey Michael, are you using the released versions of these Arrow artifacts? I
recall there being complexity to building the multi-native-binary jar file...
>> >
>> > On Tue, Jan 5, 2021 at 8:11 AM Michael Mior <mmior@apache.org> wrote:
>> >>
>> >> I am working on integrating Arrow and Gandiva with the Apache Calcite
>> >> query processing framework so Calcite is able to process SQL queries
>> >> using Arrow data sources. The basics are working well but I'm having
>> >> trouble getting a reproducible development environment set up.
>> >>
>> >> For example, on a Linux Travis CI right now, I get the following error:
>> >>
>> >> java.lang.RuntimeException: libgandiva_jni.so was not found inside JAR.
>> >>
>> >> I get something similar with AppVeyor on Windows:
>> >>
>> >> java.lang.RuntimeException: gandiva_jni.dll was not found inside JAR.
>> >>
>> >> Starting with a fresh clone of the repository, on my macOS machine I
>> >> use for development, I get the following which seems to be related to
>> >> a protobuf version mismatch:
>> >>
>> >> java.lang.UnsatisfiedLinkError:
>> >> /private/var/folders/0h/n0jrz4qn5x37gj25sxx4z2wm0000gp/T/libgandiva_jni.dylib43feded5-f265-4a6d-b1f7-2aa56e94f392:
>> >> dlopen(/private/var/folders/0h/n0jrz4qn5x37gj25sxx4z2wm0000gp/T/libgandiva_jni.dylib43feded5-f265-4a6d-b1f7-2aa56e94f392,
>> >> 1): Library not loaded:
>> >> /usr/local/opt/protobuf/lib/libprotobuf.24.dylib
>> >>       Referenced from:
>> >> /private/var/folders/0h/n0jrz4qn5x37gj25sxx4z2wm0000gp/T/libgandiva_jni.dylib43feded5-f265-4a6d-b1f7-2aa56e94f392
>> >>
>> >> I have dependencies declared on arrow-memory-netty, arrow-vector, and
>> >> arrow-gandiva but this doesn't seem to be sufficient. Any ideas what
>> >> I'm missing here?
>> >>
>> >> --
>> >> Michael Mior
>> >> mmior@apache.org

Mime
View raw message