arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julien Le Dem <jul...@dremio.com>
Subject Re: Java-C++ integration tests -- on the home stretch
Date Fri, 09 Dec 2016 22:14:19 GMT
Woot!
🎉

On Fri, Dec 9, 2016 at 2:07 PM, Wes McKinney <wesmckinn@gmail.com> wrote:

> We just got the integration test suite (binary compatibility between
> Java and C++) passing in Travis CI today!
>
> https://travis-ci.org/wesm/arrow/builds/182725476
>
> Big team effort, congrats on all the hard work!
>
> On Fri, Dec 2, 2016 at 10:50 AM, Wes McKinney <wesmckinn@gmail.com> wrote:
> > We're close to having the integration tests all passing -- Julien and
> > I have been hammering out the lingering nuances between the Java and
> > C++ implementations. There's a number of JIRAs remaining linked to
> > from this issue:
> >
> > https://github.com/apache/arrow/pull/219
> >
> > On Mon, Nov 21, 2016 at 8:55 PM, Wes McKinney <wesmckinn@gmail.com>
> wrote:
> >> hey Ted
> >>
> >> On Mon, Nov 21, 2016 at 8:20 PM, Ted Dunning <ted.dunning@gmail.com>
> wrote:
> >>> Wes,
> >>>
> >>> This is awesome.
> >>>
> >>> Does it, however, imply that to run the tests that a C programmer will
> need
> >>> a working Java environment and a Java programmer will need a C
> environment?
> >>>
> >>> Is there any way around that? Possibly by storing golden bits for the
> >>> in-memory images somewhere?
> >>>
> >>
> >> Easiest thing would be to create a Dockerfile for experimentation --
> >> this would be useful for benchmarking on different hardware
> >> environments as well. We'll want to run the integration tests either
> >> in Travis CI or Circle CI anyway (right now we have the Java and
> >> C++/Python unit tests running in separate build setups in Travis CI),
> >> so it hopefully wouldn't be a great deal of additional effort to put
> >> everything into a container recipe.
> >>
> >> Wes
> >>
> >>>
> >>>
> >>>
> >>>
> >>> On Mon, Nov 21, 2016 at 2:05 PM, Wes McKinney <wesmckinn@gmail.com>
> wrote:
> >>>
> >>>> hi folks,
> >>>>
> >>>> After a long road, we're getting very close to having tests proving
> >>>> that the Java and C++ Arrow implementations are binary compatible --
> >>>> this will be an exciting major milestone for the project. If you
> >>>> haven't been following along recent JIRAs, the way these tests work
is
> >>>> as follows:
> >>>>
> >>>> 1) Testing dataset is specified in JSON format
> >>>>
> >>>> 2) Producer library (e.g. Java) reads JSON into Arrow in-memory, then
> >>>> writes out to an Arrow file IPC binary format
> >>>>
> >>>> 3) Consumer library (e.g. C++) attempts to read both the JSON and the
> >>>> binary file yielded by the producer library. The consumer compares the
> >>>> in-memory schemas and columnar data structures and indicates whether
> >>>> they are binary-identical
> >>>>
> >>>> I found a couple initial incompatibilities in the file format
> >>>> implementations, cited here:
> >>>> https://github.com/apache/arrow/pull/211#issuecomment-262080545.
> >>>>
> >>>> Thanks
> >>>> Wes
> >>>>
>



-- 
Julien

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message