arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Cutler <cutl...@gmail.com>
Subject Re: Java-C++ integration tests -- on the home stretch
Date Mon, 12 Dec 2016 17:44:28 GMT
This is great, nice job guys!

On Fri, Dec 9, 2016 at 2:15 PM, Jason Altekruse <altekrusejason@gmail.com>
wrote:

> Congrats guys, great work!
>
> On Fri, Dec 9, 2016 at 2:14 PM, Julien Le Dem <julien@dremio.com> wrote:
>
> > Woot!
> > 🎉
> >
> > On Fri, Dec 9, 2016 at 2:07 PM, Wes McKinney <wesmckinn@gmail.com>
> wrote:
> >
> > > We just got the integration test suite (binary compatibility between
> > > Java and C++) passing in Travis CI today!
> > >
> > > https://travis-ci.org/wesm/arrow/builds/182725476
> > >
> > > Big team effort, congrats on all the hard work!
> > >
> > > On Fri, Dec 2, 2016 at 10:50 AM, Wes McKinney <wesmckinn@gmail.com>
> > wrote:
> > > > We're close to having the integration tests all passing -- Julien and
> > > > I have been hammering out the lingering nuances between the Java and
> > > > C++ implementations. There's a number of JIRAs remaining linked to
> > > > from this issue:
> > > >
> > > > https://github.com/apache/arrow/pull/219
> > > >
> > > > On Mon, Nov 21, 2016 at 8:55 PM, Wes McKinney <wesmckinn@gmail.com>
> > > wrote:
> > > >> hey Ted
> > > >>
> > > >> On Mon, Nov 21, 2016 at 8:20 PM, Ted Dunning <ted.dunning@gmail.com
> >
> > > wrote:
> > > >>> Wes,
> > > >>>
> > > >>> This is awesome.
> > > >>>
> > > >>> Does it, however, imply that to run the tests that a C programmer
> > will
> > > need
> > > >>> a working Java environment and a Java programmer will need a C
> > > environment?
> > > >>>
> > > >>> Is there any way around that? Possibly by storing golden bits
for
> the
> > > >>> in-memory images somewhere?
> > > >>>
> > > >>
> > > >> Easiest thing would be to create a Dockerfile for experimentation
--
> > > >> this would be useful for benchmarking on different hardware
> > > >> environments as well. We'll want to run the integration tests either
> > > >> in Travis CI or Circle CI anyway (right now we have the Java and
> > > >> C++/Python unit tests running in separate build setups in Travis
> CI),
> > > >> so it hopefully wouldn't be a great deal of additional effort to put
> > > >> everything into a container recipe.
> > > >>
> > > >> Wes
> > > >>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Mon, Nov 21, 2016 at 2:05 PM, Wes McKinney <wesmckinn@gmail.com
> >
> > > wrote:
> > > >>>
> > > >>>> hi folks,
> > > >>>>
> > > >>>> After a long road, we're getting very close to having tests
> proving
> > > >>>> that the Java and C++ Arrow implementations are binary compatible
> --
> > > >>>> this will be an exciting major milestone for the project.
If you
> > > >>>> haven't been following along recent JIRAs, the way these tests
> work
> > is
> > > >>>> as follows:
> > > >>>>
> > > >>>> 1) Testing dataset is specified in JSON format
> > > >>>>
> > > >>>> 2) Producer library (e.g. Java) reads JSON into Arrow in-memory,
> > then
> > > >>>> writes out to an Arrow file IPC binary format
> > > >>>>
> > > >>>> 3) Consumer library (e.g. C++) attempts to read both the JSON
and
> > the
> > > >>>> binary file yielded by the producer library. The consumer
compares
> > the
> > > >>>> in-memory schemas and columnar data structures and indicates
> whether
> > > >>>> they are binary-identical
> > > >>>>
> > > >>>> I found a couple initial incompatibilities in the file format
> > > >>>> implementations, cited here:
> > > >>>> https://github.com/apache/arrow/pull/211#issuecomment-262080545.
> > > >>>>
> > > >>>> Thanks
> > > >>>> Wes
> > > >>>>
> > >
> >
> >
> >
> > --
> > Julien
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message