From user-return-189-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Sat Aug 31 11:53:04 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 4B59018063F for ; Sat, 31 Aug 2019 13:53:04 +0200 (CEST) Received: (qmail 30658 invoked by uid 500); 31 Aug 2019 11:53:03 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 30647 invoked by uid 99); 31 Aug 2019 11:53:03 -0000 Received: from Unknown (HELO mailrelay1-lw-us.apache.org) (10.10.3.159) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 31 Aug 2019 11:53:03 +0000 Received: from mail-ot1-f41.google.com (mail-ot1-f41.google.com [209.85.210.41]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id 5D585599F for ; Sat, 31 Aug 2019 11:53:03 +0000 (UTC) Received: by mail-ot1-f41.google.com with SMTP id n7so2039164otk.6 for ; Sat, 31 Aug 2019 04:53:03 -0700 (PDT) X-Gm-Message-State: APjAAAVSVoWK1wLNdrx8/KbfMTB40IcUDiQNC0R/leYox2buu13XghCJ xO66SmvsEp5Q1sASwwXNU8XlloFib9/aGbjmtK0= X-Google-Smtp-Source: APXvYqxJvjbqrdLwf14oILj3e8l4/KXnbS54A2nzGy19zkVIwKR8YGblihBG6m5WQ+d/H3QaINIsBLsw0nSiHw45T1U= X-Received: by 2002:a9d:17cb:: with SMTP id j69mr6312946otj.69.1567252382771; Sat, 31 Aug 2019 04:53:02 -0700 (PDT) MIME-Version: 1.0 References: <16ce1f3a1c0.27b7.727fde9afb6860196ef19c74546e60d2@ipp.mpg.de> <16ce3e95f10.27b7.727fde9afb6860196ef19c74546e60d2@ipp.mpg.de> In-Reply-To: From: Jacques Nadeau Date: Sat, 31 Aug 2019 04:52:49 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Reccord-Level Access To: user@arrow.apache.org, Micah Kornfield Content-Type: multipart/alternative; boundary="0000000000008c6f8505916861b5" --0000000000008c6f8505916861b5 Content-Type: text/plain; charset="UTF-8" I'm arrow Java there is a record level accesor facade called FieldReader. There is also a record level builder called ComplexWriter. Both allow arbitrary complex data to be worked with using method invocations. Won't be as efficient as a columnar algorithm but definitely much easier to get started. We use a heavily overloaded interface pattern to basically support dynamic typing. Example reader use https://github.com/dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/exec/vector/complex/fn/JsonWriter.java Example builder use https://github.com/dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/exec/vector/complex/fn/JsonReader.java On Fri, Aug 30, 2019, 8:14 PM Micah Kornfield wrote: > Hi Simon, > A couple notes: > - Scalars are a C++ thing, There are ValueHolders in Java but i'm not sure > what you want. > - Dremio has a JDBC adaptor [1] that might be worth looking at (or maybe > porting pieces of it into Arrow). > > Thanks, > Micah > > [1] > https://github.com/dremio/dremio-oss/blob/04e0387d474f1408731da0029aef7ecfad5e4d08/client/jdbc/src/main/java/com/dremio/jdbc/impl/DremioResultSetImpl.java > > > > On Fri, Aug 30, 2019 at 12:03 PM Simon Dumke > wrote: > >> Hi Ben, >> >> thanks for the suggestion, i'll loon into it! >> >> Regards, >> Simon >> >> Am 30. August 2019 20:59:23 schrieb Ben Kietzman < >> ben.kietzman@rstudio.com>: >> >>> Hi Simon, >>> >>> If you're interested in adding a record interface, the Scalar classes >>> might be a good place to start. They represent a value from an array slot >>> and it should be fairly straightforward to extract a table row as a >>> StructScalar >>> >>> On Fri, Aug 30, 2019 at 1:27 PM Wes McKinney >>> wrote: >>> >>>> hi Simon -- I don't think there is any such Row accessor class in Java >>>> but you are welcome to contribute one to the project. For performance >>>> sensitive applications, using a record interface might not be the best >>>> idea, but I can understand the convenience for some uses cases. >>>> >>>> - Wes >>>> >>>> On Fri, Aug 30, 2019 at 4:55 AM Simon Dumke >>>> wrote: >>>> > >>>> > Hi all, >>>> > >>>> > >>>> > >>>> > >>>> > I did not find anything (and so: no definite answer) in the docs, so i >>>> > thought to ask here: >>>> > >>>> > >>>> > >>>> > >>>> > Does Arrow (and at this point my main concern is Arrow for java) >>>> support >>>> > any type of concept that allows a "record level access" (so, a "row") >>>> to >>>> > data in an Arrow RecordBatch or Table? I would have thougt that even >>>> in >>>> > column-oriented analytics etc. this would be a common last step access >>>> > pattern over many use cases, but i could not find any references to >>>> such a >>>> > thing. >>>> > >>>> > >>>> > >>>> > >>>> > Thanks and kind regards, >>>> > Simon >>>> > >>>> > >>>> >>> >> --0000000000008c6f8505916861b5 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I'm arrow Java there is a record level accesor facade= called FieldReader. There is also a record level builder called ComplexWri= ter. Both allow arbitrary complex data to be worked with using method invoc= ations. Won't be as efficient as a columnar algorithm but definitely mu= ch easier to get started. We use a heavily overloaded interface pattern to = basically support dynamic typing.=C2=A0

On Fri, Aug 30, 2019, 8:14 PM M= icah Kornfield <emkornfield@gma= il.com> wrote:
Hi Simon,
A couple notes:
- Scalars are a C++ thing= , There are ValueHolders in Java but i'm not sure what you want.
- = Dremio has a JDBC adaptor [1] that might be worth looking at (or maybe port= ing pieces of it into Arrow).

On = Fri, Aug 30, 2019 at 12:03 PM Simon Dumke <simon.dumke@ipp.mpg.de> wrote:
=
Hi Ben,

thanks=C2=A0for the suggestion, i'll l= oon into it!
<= br>
Regards,
Simon

Am 30. August 2019 20:59:23 schrieb Ben Kietzman <ben.kietzman@= rstudio.com>:

Hi Simon,

If you're inte= rested in adding a record interface, the Scalar classes might be a good pla= ce to start. They represent a value from an array slot and it should be fai= rly straightforward to extract a table row as a StructScalar

On Fr= i, Aug 30, 2019 at 1:27 PM Wes McKinney <wesmckinn@gmail.com> wr= ote:
hi Simon --= I don't think there is any such Row accessor class in Java
but you are welcome to contribute one to the project. For performance
sensitive applications, using a record interface might not be the best
idea, but I can understand the convenience for some uses cases.

- Wes

On Fri, Aug 30, 2019 at 4:55 AM Simon Dumke <simon.dumke@ipp.mpg.de<= /a>> wrote:
>
> Hi all,
>
>
>
>
> I did not find anything (and so: no definite answer) in the docs, so i=
> thought to ask here:
>
>
>
>
> Does Arrow (and at this point my main concern is Arrow for java) suppo= rt
> any type of concept that allows a "record level access" (so,= a "row") to
> data in an Arrow RecordBatch or Table? I would have thougt that even i= n
> column-oriented analytics etc. this would be a common last step access=
> pattern over many use cases, but i could not find any references to su= ch a
> thing.
>
>
>
>
> Thanks and kind regards,
> Simon
>
>

--0000000000008c6f8505916861b5--