arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacques Nadeau <jacq...@apache.org>
Subject Re: Arrow/Gandiva dependency management in Java
Date Tue, 05 Jan 2021 18:09:39 GMT
If I recall correctly, our artifact doesn't support windows (just mac and
linux). If AppVeyor is for windows I would assume that would fail.

Let's see what happens with Travis. I'd expect it to succeed.

I suggest you open a Arrow GH issue to track this problem and we continue
to record the result of items there as well.

Can you also confirm the difference of the set of native libraries in the
Dremio jar versus the Apache one?




On Tue, Jan 5, 2021 at 9:49 AM Michael Mior <michael.mior@gmail.com> wrote:

> Thanks Jaques! Using the Dremio artifacts do seem to work on my macOS
> development machine but failed with the same error on AppVeyor. (Still
> waiting on Travis.) If there's anything I can do to help with
> debugging, please let me know! You can see a (somewhat messy) PR in
> progress at the link below. The last commit on the branch switches to
> the Dremio repository if you want to try both and see if you can
> reproduce (./gradlew :arrow:test should succeed if things are working
> correctly).
>
> https://github.com/apache/calcite/pull/2133
>
> --
> Michael Mior
> michael.mior@gmail.com
>
> Le mar. 5 janv. 2021 à 12:08, Jacques Nadeau <jacques@apache.org> a écrit
> :
> >
> > If you're up for troubleshooting, do you want to try the near master jar
> we use inside of Dremio? I think we're the primary users of the Gandiva
> Java bindings and I wonder if the Apache build has an issue with artifact
> packaging(?). The jar we use in Dremio is used at hundreds of customers so
> it should work without extra items. You can find the reference in the pom
> here:
> >
> > https://github.com/dremio/dremio-oss/blob/master/pom.xml#L41
> >
> > I think you'll need to add our public repo to get the jar. I think it is
> this repo:
> > https://github.com/dremio/dremio-oss/blob/master/pom.xml#L3243
> >
> > I'm only suggesting this to see if there is some kind of disconnect in
> keeping the Apache artifact complete. If our jar works for you we'll need
> to figure out what is different between it and the Apache one. (For
> reference, in general our jars are slightly off master to pull forward or
> backward certain bugs, etc.)
> >
> >
> >
> > On Tue, Jan 5, 2021 at 8:55 AM Michael Mior <mmior@apache.org> wrote:
> >>
> >> Yes, I'm using the latest released version (2.0.0).
> >> --
> >> Michael Mior
> >> mmior@apache.org
> >>
> >> Le mar. 5 janv. 2021 à 11:45, Jacques Nadeau <jacques@apache.org> a
> écrit :
> >> >
> >> > Hey Michael, are you using the released versions of these Arrow
> artifacts? I recall there being complexity to building the
> multi-native-binary jar file...
> >> >
> >> > On Tue, Jan 5, 2021 at 8:11 AM Michael Mior <mmior@apache.org> wrote:
> >> >>
> >> >> I am working on integrating Arrow and Gandiva with the Apache Calcite
> >> >> query processing framework so Calcite is able to process SQL queries
> >> >> using Arrow data sources. The basics are working well but I'm having
> >> >> trouble getting a reproducible development environment set up.
> >> >>
> >> >> For example, on a Linux Travis CI right now, I get the following
> error:
> >> >>
> >> >> java.lang.RuntimeException: libgandiva_jni.so was not found inside
> JAR.
> >> >>
> >> >> I get something similar with AppVeyor on Windows:
> >> >>
> >> >> java.lang.RuntimeException: gandiva_jni.dll was not found inside JAR.
> >> >>
> >> >> Starting with a fresh clone of the repository, on my macOS machine
I
> >> >> use for development, I get the following which seems to be related
to
> >> >> a protobuf version mismatch:
> >> >>
> >> >> java.lang.UnsatisfiedLinkError:
> >> >>
> /private/var/folders/0h/n0jrz4qn5x37gj25sxx4z2wm0000gp/T/libgandiva_jni.dylib43feded5-f265-4a6d-b1f7-2aa56e94f392:
> >> >>
> dlopen(/private/var/folders/0h/n0jrz4qn5x37gj25sxx4z2wm0000gp/T/libgandiva_jni.dylib43feded5-f265-4a6d-b1f7-2aa56e94f392,
> >> >> 1): Library not loaded:
> >> >> /usr/local/opt/protobuf/lib/libprotobuf.24.dylib
> >> >>       Referenced from:
> >> >>
> /private/var/folders/0h/n0jrz4qn5x37gj25sxx4z2wm0000gp/T/libgandiva_jni.dylib43feded5-f265-4a6d-b1f7-2aa56e94f392
> >> >>
> >> >> I have dependencies declared on arrow-memory-netty, arrow-vector, and
> >> >> arrow-gandiva but this doesn't seem to be sufficient. Any ideas what
> >> >> I'm missing here?
> >> >>
> >> >> --
> >> >> Michael Mior
> >> >> mmior@apache.org
>

Mime
View raw message