arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emilio Lahr-Vivaz <>
Subject Re: buffer alignment (format/java/js)
Date Tue, 08 Aug 2017 12:52:03 GMT
After looking at it further, I think only the buffers themselves need to 
be aligned, not the metadata and/or schema. Would there be any problem 
with changing the alignment to 64 bytes then?



On 08/08/2017 08:08 AM, Emilio Lahr-Vivaz wrote:
> I'm looking into buffer alignment in the java writer classes. 
> Currently some files written with the java streaming writer can't be 
> read due to the javascript TypedArray's restriction that the start 
> offset of the array must be a multiple of the data size of the array 
> type (i.e. Int32Vectors must start on a multiple of 4, Float64Vectors 
> must start on a multiple of 8, etc). From a cursory look at the java 
> writer, I believe that the schema that is written first is not aligned 
> at all, and then each record batch pads out its size to a multiple of 
> 8. So:
> 1. should the schema block pad itself so that the first record batch 
> is aligned, and is there any problem with doing so?
> 2. is there any problem with changing the alignment to 64 bytes, as 
> recommended (but not required) by the spec?
> Thanks,
> Emilio

View raw message