Mailing-List: contact dev-help@arrow.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@arrow.apache.org
Subject: Re: buffer alignment (format/java/js)
To: dev@arrow.apache.org
References: <ca583b35-f376-fc9a-54e0-16ae6e949ffe@ccri.com>
From: Emilio Lahr-Vivaz <elahrvivaz@ccri.com>
Message-ID: <e9c906ba-9165-52eb-22cb-3f4eaf4be062@ccri.com>
Date: Tue, 8 Aug 2017 08:52:03 -0400
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
 Thunderbird/52.2.1
MIME-Version: 1.0
In-Reply-To: <ca583b35-f376-fc9a-54e0-16ae6e949ffe@ccri.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Content-Language: en-US
archived-at: Tue, 08 Aug 2017 12:52:17 -0000

After looking at it further, I think only the buffers themselves need to 
be aligned, not the metadata and/or schema. Would there be any problem 
with changing the alignment to 64 bytes then?

Thanks,

Emilio

On 08/08/2017 08:08 AM, Emilio Lahr-Vivaz wrote:
> I'm looking into buffer alignment in the java writer classes. 
> Currently some files written with the java streaming writer can't be 
> read due to the javascript TypedArray's restriction that the start 
> offset of the array must be a multiple of the data size of the array 
> type (i.e. Int32Vectors must start on a multiple of 4, Float64Vectors 
> must start on a multiple of 8, etc). From a cursory look at the java 
> writer, I believe that the schema that is written first is not aligned 
> at all, and then each record batch pads out its size to a multiple of 
> 8. So:
>
> 1. should the schema block pad itself so that the first record batch 
> is aligned, and is there any problem with doing so?
> 2. is there any problem with changing the alignment to 64 bytes, as 
> recommended (but not required) by the spec?
>
> Thanks,
>
> Emilio