arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wes McKinney <wesmck...@gmail.com>
Subject Completing additional Arrow Java/C++ integration tests
Date Mon, 22 May 2017 21:26:22 GMT
hi folks,

We've come a long way in our integration testing in the last 6 months.
For those who haven't followed actively, the integration tests verify
wire compatibility between Java and C++ streaming and file binary
formats. We run these tests continuously in Travis CI, which makes it
safe for developers to refactor and improve the IPC / wire format code
without breaking cross-implementation compatibility. I expect in the
near future we will also see some JavaScript integration tests.

The integration test suite works as follows:

* Test data is specified in an Arrow-specific JSON format
* Each application reads the JSON and converts it to the Arrow binary
format according to its implementation
* The other application reads the binary and compares the binary to
the JSON "point of truth" file

What is still missing from the current test suite:

- Dictionary tests
- Fixed size list
- Decimal tests (first available in C++ in 0.3.0)
- Unions
- Intervals

Otherwise we have pretty good coverage of the rest of the data types
in https://github.com/apache/arrow/blob/master/format/Schema.fbs; a
lot of work in Arrow 0.3.0 was reconciling the date and time types.

For the next release 0.5.0 the most important thing to test is
dictionaries. It would be nice to test decimals also, but we have some
specification work to do as the in-memory format for Decimals in C++
and Java is currently different.

I should have time to implement the C++ side of the dictionary
integration tests (i.e. converting to/from the JSON format), but
someone will need to implement the Java side. It would be great if we
could get this done in the next few weeks if anyone has the time to do
it.

Thanks
Wes

Mime
View raw message