arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wes McKinney (JIRA)" <>
Subject [jira] [Created] (ARROW-57) Format: Draft data headers IDL for data interchange
Date Thu, 10 Mar 2016 00:59:40 GMT
Wes McKinney created ARROW-57:

             Summary: Format: Draft data headers IDL for data interchange
                 Key: ARROW-57
             Project: Apache Arrow
          Issue Type: New Feature
          Components: Format
            Reporter: Wes McKinney

>From the mailing list discussion, we need to define the structure of data headers to describe
a "page" of Arrow data. In Apache Parquet these are referred to as the data page headers and
are separate from the schema metadata.

As part of this JIRA:

* Choose a technology (it need not be the only one, but it would be good to have a reference
tool) for representing the data page header in "serialized" form (as can be stored in shared
memory and interpreted by any system with access to that memory). Google's Flatbuffers libraries
has been discussed for this use case.

* Draft IDL specification for the data headers

This message was sent by Atlassian JIRA

View raw message