arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wes McKinney <wesmck...@gmail.com>
Subject Re: What's the best way to construct a arrow record batch for testing/validation in Java?
Date Wed, 07 Dec 2016 17:43:47 GMT
hi Li,

This is exactly what we are doing in the integration tests. See the
"JSON_TO_ARROW" and "VALIDATE" commands in the Java integration tests:

https://github.com/apache/arrow/blob/master/java/tools/src/main/java/org/apache/arrow/tools/Integration.java

Here is a sample JSON data file:

https://github.com/apache/arrow/blob/master/integration/data/simple.json

In my patch in https://github.com/apache/arrow/pull/219 I started
creating more comprehensive JSON data generation, so you will be able
to more easily generate a JSON file that matches a particular record
batch schema.

- Wes

On Wed, Dec 7, 2016 at 11:00 AM, Li Jin <ice.xelloss@gmail.com> wrote:
> Hello!
>
> I am trying to test a function that turns a list of some data to a arrow
> record batch. In order to do that, I need to compare the output of the
> function to a "correct" arrow record batch. However, I struggle with
> creating the "correct" arrow record batch.
>
> My test data is a list of rows that each has a integer column "a" with a
> null value:
> [{a: 1}, {a: 2}, {a: 3}, {a: null}]
>
> Is there a format of data that I can use to turn into arrow record batch
> easily (for instance, json?) ?

Mime
View raw message