From user-return-190-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Sat Aug 31 15:21:19 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id A293E18063F for ; Sat, 31 Aug 2019 17:21:19 +0200 (CEST) Received: (qmail 74784 invoked by uid 500); 31 Aug 2019 15:21:18 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 74774 invoked by uid 99); 31 Aug 2019 15:21:18 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 31 Aug 2019 15:21:18 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 27F8B1805F1 for ; Sat, 31 Aug 2019 15:21:18 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.801 X-Spam-Level: * X-Spam-Status: No, score=1.801 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 3pcf9cSEyvgt for ; Sat, 31 Aug 2019 15:21:16 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.208.179; helo=mail-lj1-f179.google.com; envelope-from=emkornfield@gmail.com; receiver= Received: from mail-lj1-f179.google.com (mail-lj1-f179.google.com [209.85.208.179]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id A45E3BC7A9 for ; Sat, 31 Aug 2019 15:21:15 +0000 (UTC) Received: by mail-lj1-f179.google.com with SMTP id u15so9106508ljl.3 for ; Sat, 31 Aug 2019 08:21:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:reply-to:in-reply-to:references:from:date:message-id :subject:to; bh=oaEw1nN3nRo74VRv++/qkdj2C5dqUfR1CiY94hJm6l0=; b=X84HyD8Tm68KEP2Q4kJk5mEQnMQXiLbF/bKHC3ewxg4PZZYm6HhQva44a3TtqGpJ2l V1EALvy43jhezi/9lt5FC1CradkfQeDfMO6UqtAXryFJ40efbxIhtzw+llVdy5OHDB69 LWZFbUtMrdrFyGE+zlxMiUeKP8cn4EFoUVL0Ce+c2nmUWDOCrK9tZXsWopux85ZxuzEa l1VPf4lB80Qh0ELDKu5lRwA/5dwPWlL02MMiO/bLAtYdi0L6jp5xbF7jGlNGD35zvQh4 E0Vgy7qIAlZSB3deoLPp07B4oBotYO5mADbvl4Vou6cmbrH7YuITGqRyrnAqZQGHAeoh UzaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :from:date:message-id:subject:to; bh=oaEw1nN3nRo74VRv++/qkdj2C5dqUfR1CiY94hJm6l0=; b=hsbG9HFtPuh/TMQwr87IWyQLRqsMck1F+gxrdLSOLTKNAA/YxJRt0YtYfQpD+bCxPx UkzczWie8+Afo4vh/139ISVNl7DqFibJfiNCMfQSBJdgNZnpMNsBpN+EHPk8oJdZwZCe dXrTnUzjdSsPr9/4WwiYmt3wHXUYpTdBhZff0DkmXOFMsVCo6hnoCslhcWDGJIN6EsMn YBOFipwkRzXGy54Es7vqvH/4/IvSgqjhEeQxu3kMgGD7bcHh3M0uog+plovwQdu/GFjv RgD/1HzdZpEu3uIY+6kry5uJlH2SgRAtUM0jiFYTf4ZAYeeWmMcABl7fR3NyePHGhBrF y1JA== X-Gm-Message-State: APjAAAUCe8zwGJINQBuDnvxZcxkRH0iOUiVhA2SX5Q6JgHB5dBQ+tt8a 3EdB3j/M931luPrnJxMCjMdTYzicYHDrtRx1Bk5rxg== X-Google-Smtp-Source: APXvYqz7uz1AtWgUMx/T0uEnmTUP/H91qvU6KC7RXUcxrNMK8UzjL6zGMl5AUaT3wO9Mdx4FSJp2xFAUUU0bFIIjZKE= X-Received: by 2002:a2e:9889:: with SMTP id b9mr11863446ljj.230.1567264873990; Sat, 31 Aug 2019 08:21:13 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:ab3:7659:0:0:0:0:0 with HTTP; Sat, 31 Aug 2019 08:21:13 -0700 (PDT) Reply-To: emkornfield@gmail.com In-Reply-To: References: <16ce1f3a1c0.27b7.727fde9afb6860196ef19c74546e60d2@ipp.mpg.de> <16ce3e95f10.27b7.727fde9afb6860196ef19c74546e60d2@ipp.mpg.de> From: Micah Kornfield Date: Sat, 31 Aug 2019 08:21:13 -0700 Message-ID: Subject: Re: Reccord-Level Access To: "user@arrow.apache.org" Content-Type: multipart/alternative; boundary="00000000000015504505916b4a40" --00000000000015504505916b4a40 Content-Type: text/plain; charset="UTF-8" That is a good point, I didn't initially think of these because they are missing an adapter or documentation (creating a top level struct) on how to make them work with a VectorSchemaRoot. On Saturday, August 31, 2019, Jacques Nadeau wrote: > I'm arrow Java there is a record level accesor facade called FieldReader. > There is also a record level builder called ComplexWriter. Both allow > arbitrary complex data to be worked with using method invocations. Won't be > as efficient as a columnar algorithm but definitely much easier to get > started. We use a heavily overloaded interface pattern to basically support > dynamic typing. > > Example reader use > https://github.com/dremio/dremio-oss/blob/master/sabot/ > kernel/src/main/java/com/dremio/exec/vector/complex/fn/JsonWriter.java > > Example builder use > https://github.com/dremio/dremio-oss/blob/master/sabot/ > kernel/src/main/java/com/dremio/exec/vector/complex/fn/JsonReader.java > > On Fri, Aug 30, 2019, 8:14 PM Micah Kornfield > wrote: > >> Hi Simon, >> A couple notes: >> - Scalars are a C++ thing, There are ValueHolders in Java but i'm not >> sure what you want. >> - Dremio has a JDBC adaptor [1] that might be worth looking at (or maybe >> porting pieces of it into Arrow). >> >> Thanks, >> Micah >> >> [1] https://github.com/dremio/dremio-oss/blob/ >> 04e0387d474f1408731da0029aef7ecfad5e4d08/client/jdbc/src/ >> main/java/com/dremio/jdbc/impl/DremioResultSetImpl.java >> >> >> >> On Fri, Aug 30, 2019 at 12:03 PM Simon Dumke >> wrote: >> >>> Hi Ben, >>> >>> thanks for the suggestion, i'll loon into it! >>> >>> Regards, >>> Simon >>> >>> Am 30. August 2019 20:59:23 schrieb Ben Kietzman < >>> ben.kietzman@rstudio.com>: >>> >>>> Hi Simon, >>>> >>>> If you're interested in adding a record interface, the Scalar classes >>>> might be a good place to start. They represent a value from an array slot >>>> and it should be fairly straightforward to extract a table row as a >>>> StructScalar >>>> >>>> On Fri, Aug 30, 2019 at 1:27 PM Wes McKinney >>>> wrote: >>>> >>>>> hi Simon -- I don't think there is any such Row accessor class in Java >>>>> but you are welcome to contribute one to the project. For performance >>>>> sensitive applications, using a record interface might not be the best >>>>> idea, but I can understand the convenience for some uses cases. >>>>> >>>>> - Wes >>>>> >>>>> On Fri, Aug 30, 2019 at 4:55 AM Simon Dumke >>>>> wrote: >>>>> > >>>>> > Hi all, >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > I did not find anything (and so: no definite answer) in the docs, so >>>>> i >>>>> > thought to ask here: >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > Does Arrow (and at this point my main concern is Arrow for java) >>>>> support >>>>> > any type of concept that allows a "record level access" (so, a >>>>> "row") to >>>>> > data in an Arrow RecordBatch or Table? I would have thougt that even >>>>> in >>>>> > column-oriented analytics etc. this would be a common last step >>>>> access >>>>> > pattern over many use cases, but i could not find any references to >>>>> such a >>>>> > thing. >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > Thanks and kind regards, >>>>> > Simon >>>>> > >>>>> > >>>>> >>>> >>> --00000000000015504505916b4a40 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable That is a good point, I didn't initially think of these because they ar= e missing an adapter or documentation (creating a top level struct) on how = to make them work with a VectorSchemaRoot.

On Saturday, August 31, 2= 019, Jacques Nadeau <jacques@apach= e.org> wrote:
I&= #39;m arrow Java there is a record level accesor facade called FieldReader.= There is also a record level builder called ComplexWriter. Both allow arbi= trary complex data to be worked with using method invocations. Won't be= as efficient as a columnar algorithm but definitely much easier to get sta= rted. We use a heavily overloaded interface pattern to basically support dy= namic typing.=C2=A0

On Fri, Aug 30, 2019, 8:14 PM Micah Kornfield &l= t;emkornfield@gm= ail.com> wrote:
Hi Simon,
A couple notes:
- Scalars are a C++ thin= g, There are ValueHolders in Java but i'm not sure what you want.
-= Dremio has a JDBC adaptor [1] that might be worth looking at (or maybe por= ting pieces of it into Arrow).

On Fri, Aug 30, 2019 at 12:03 PM Simon Dumke <simon.dumke@ipp.mpg.de> wrote:
Hi Ben,

thanks=C2=A0for the suggestion, i'll l= oon into it!
<= br>
Regards,
Simon

Am 30. August 2019 20:59:23 schrieb Ben Kietzman <ben.kietzman@= rstudio.com>:

Hi Simon,

If you're inte= rested in adding a record interface, the Scalar classes might be a good pla= ce to start. They represent a value from an array slot and it should be fai= rly straightforward to extract a table row as a StructScalar

On Fr= i, Aug 30, 2019 at 1:27 PM Wes McKinney <wesmckinn@gmail.com> wr= ote:
hi Simon --= I don't think there is any such Row accessor class in Java
but you are welcome to contribute one to the project. For performance
sensitive applications, using a record interface might not be the best
idea, but I can understand the convenience for some uses cases.

- Wes

On Fri, Aug 30, 2019 at 4:55 AM Simon Dumke <simon.dumke@ipp.mpg.de<= /a>> wrote:
>
> Hi all,
>
>
>
>
> I did not find anything (and so: no definite answer) in the docs, so i=
> thought to ask here:
>
>
>
>
> Does Arrow (and at this point my main concern is Arrow for java) suppo= rt
> any type of concept that allows a "record level access" (so,= a "row") to
> data in an Arrow RecordBatch or Table? I would have thougt that even i= n
> column-oriented analytics etc. this would be a common last step access=
> pattern over many use cases, but i could not find any references to su= ch a
> thing.
>
>
>
>
> Thanks and kind regards,
> Simon
>
>

--00000000000015504505916b4a40--