arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wes McKinney (JIRA)" <j...@apache.org>
Subject [jira] [Created] (ARROW-76) Revise format document to include null count, defer non-nullable arrays to the domain of metadata
Date Wed, 23 Mar 2016 02:01:25 GMT
Wes McKinney created ARROW-76:
---------------------------------

             Summary: Revise format document to include null count, defer non-nullable arrays
to the domain of metadata
                 Key: ARROW-76
                 URL: https://issues.apache.org/jira/browse/ARROW-76
             Project: Apache Arrow
          Issue Type: New Feature
          Components: Format
            Reporter: Wes McKinney


After many mailing list discussions and a first draft of metadata serialization (https://github.com/apache/arrow/blob/master/format/Message.fbs),
I propose we strike the notion of required fields from the physical memory layout and instead
make this a property of the metadata (see the {{nullable}} member of {{Field}} in the serialized
schema metadata). 

Note that Arrow implementations will be free to create specialized containers for data having
{{nullable = false}} in their metadata. It will be the implementations' responsibility to
validate that {{null_count == 0}} at IPC boundaries when interacting with "untrusted" data.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message