Mailing-List: contact dev-help@arrow.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@arrow.apache.org
MIME-Version: 1.0
In-Reply-To: <CAK7Z5T_9kasHH55uGXwDYUupm3e9Nsa7xF22B9HZYQ_E2KPMiQ@mail.gmail.com>
References: <CAK7Z5T-6CZc1UXGso0uJpkcjqdv5cztUKD8oLE+ezVYx7b6SWQ@mail.gmail.com>
 <CAJPUwMAHd-MLMU-+8bubWUTBYX6_-Wm8TLsiJ_5HdVhOY8JwxQ@mail.gmail.com>
 <CAKa9qD=_WeD8HBa=1qdsVOyjDQCUX4ui-bwN1VO5xqqYaV640Q@mail.gmail.com>
 <CAK7Z5T9X=Vvk22qBeciJ0icHj7UiWk1z9DU+3u5-R-Tq3js=6Q@mail.gmail.com>
 <CAK7Z5T9W3XSw+8=v-=Abz17fufTDsOf1o2Rwkx3n4Q0jb6wY5g@mail.gmail.com>
 <CAJPUwMDqWoQjGzzQQhJSM=4K1PO-i+wO5fuOss7ChfFmpqyOmw@mail.gmail.com>
 <CAK7Z5T-WGRbx4OQqBmr1n2-gtfMgNLVXeV=fNNjTHUEjpMu-yA@mail.gmail.com>
 <CAJPUwMDxMx=NhTavcGH0tmKujdxQ6d7TUVCa5N47sdCf2GiibA@mail.gmail.com> <CAK7Z5T_9kasHH55uGXwDYUupm3e9Nsa7xF22B9HZYQ_E2KPMiQ@mail.gmail.com>
From: Julien Le Dem <julien@dremio.com>
Date: Mon, 15 Aug 2016 14:04:58 -0700
Message-ID: <CAO+vc4C7xrcM=usg0fsBwfG7ahF_57jP6+c7PMS+mFf6oCKP4Q@mail.gmail.com>
Subject: Re: Discussion: Should we make string/binary types first class Arrow
 Array types?
To: Micah Kornfield <emkornfield@gmail.com>
Cc: Wes McKinney <wesmckinn@gmail.com>, dev@arrow.apache.org,
	Jacques Nadeau <jacques@apache.org>
Content-Type: multipart/alternative; boundary=001a1143f7aef132c5053a229671
archived-at: Mon, 15 Aug 2016 21:05:25 -0000

--001a1143f7aef132c5053a229671
Content-Type: text/plain; charset=UTF-8

There's ARROW-258 which is about clarifying difference (if any) in metadata
across RPC (sockets), IPC (shared memory) and files.
The vector layout is the same except in RPC or files they get concatenated
together when copied over.
The metadata should be mostly the same (ideally the same). Buffer offsets
are relative to the beginning of the body in the context of RPC and file
start in files. In the context of IPC it looks like we need an extra page
id (from Message.fbs). Is this correct?

On Mon, Aug 15, 2016 at 12:01 PM, Micah Kornfield <emkornfield@gmail.com>
wrote:

> Thanks Wes,
> This makes sense.  +1 on the "Logical Types / IPC layout
> document"  is there a JIRA open for this?
>
> I'll open a JIRA item to change the inheritance of string/binary in the
> C++ code base.
>
> Thanks,
> Micah
>
> On Sun, Aug 14, 2016 at 10:51 PM, Wes McKinney <wesmckinn@gmail.com>
> wrote:
>
>> On Fri, Aug 12, 2016 at 5:57 PM, Micah Kornfield <emkornfield@gmail.com>
>> wrote:
>> > Sorry for the late reply.
>> >
>> > This all sounds reasonable to me.  But I'm not sure I understand exactly
>> > what you mean by
>> >
>> >> Accordingly, in the metadata and in RPC/IPC scenarios, binary/string
>> >> would be a single array unit in the buffer stream and flattened Field
>> >> metadata rather than nested types (2 array units as they are
>> >> presently).
>> >
>> >
>> > The way I read it this seems to me to contradict the
>> cross-implementation as
>> > "List<UInt8-not null>"?
>> >
>> > Thanks,
>> > Micah
>> >
>>
>> I think we can resolve this by starting a "Logical Types and IPC/RPC
>> layout" specification document.
>>
>> The schema metadata
>> (https://github.com/apache/arrow/blob/master/format/Message.fbs) is,
>> as I understand it, strictly the domain of logical types. I believe
>> there is some minor conflation of the notions of primitive physical
>> types and primitive logical types.
>>
>> While String / Binary have identical physical layouts to List<UInt8
>> not null>, in the domain of logical types and IPC, what we are saying
>> is that these types are:
>>
>> - logically speaking: primitive, non-nested types
>> - their IPC layout is the flattened version of the nested List<UInt8>
>> counterpart -- a single Field node having String type (with a null
>> count, etc.), and 3 memory buffers: validity bitmap, offsets, and
>> data. Structurally on the wire / in shared memory (compared with
>> List<UInt8 not null>) the only difference is the Field metadata (since
>> if null count is 0 for the inner UInt8 values, then there is only a
>> single buffer) -- one node versus two
>>
>> Let me know if this does not make sense.
>>
>> To move this forward I propose to begin a Logical Types / IPC layout
>> document and begin to document the mapping between logical types and
>> their physical in-memory representation and layout on the wire.
>>
>> - Wes
>>
>
>


-- 
Julien

--001a1143f7aef132c5053a229671--