arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julien Le Dem (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ARROW-253) Int types should only have width of 8*2^n (8, 16, 32, 64)
Date Wed, 10 Aug 2016 21:44:20 GMT

    [ https://issues.apache.org/jira/browse/ARROW-253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15416071#comment-15416071
] 

Julien Le Dem commented on ARROW-253:
-------------------------------------

I'm familiar with the BitWeaving techniques and we should certainly look into that in the
future. 
I'm not against bit packed vectors, I'm just wondering if they should be in V1

The other aspect we should consider is versioning of the spec and how we add more vectors
laters. We want to get to a point soon where all of the spec is supported so we have to draw
a line somewhere and limit the surface area for implementors to ease adoption. Nothing prevents
people to experiment with other representations (as long as they stay internal). Those can
be added to the spec later.

some options: 
 - restrict int size in Arrow V1. Add bitpacked int vectors in V2.
 - leave all widths (1 to 64) available in V1. Add converters to efficiently unpack for engines
that don't operate on packed vectors. (but someone needs to create those).



> Int types should only have width of 8*2^n (8, 16, 32, 64)
> ---------------------------------------------------------
>
>                 Key: ARROW-253
>                 URL: https://issues.apache.org/jira/browse/ARROW-253
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: Format
>            Reporter: Julien Le Dem
>
> Since arrow is focused on CPU efficient operations and we want to keep the spec simple
to the minimum required we should restrict the Int type to width of 1, 2, 4 or 8 bytes



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message