arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacques Nadeau <jacq...@apache.org>
Subject Re: Reccord-Level Access
Date Sat, 31 Aug 2019 11:52:49 GMT
I'm arrow Java there is a record level accesor facade called FieldReader.
There is also a record level builder called ComplexWriter. Both allow
arbitrary complex data to be worked with using method invocations. Won't be
as efficient as a columnar algorithm but definitely much easier to get
started. We use a heavily overloaded interface pattern to basically support
dynamic typing.

Example reader use
https://github.com/dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/exec/vector/complex/fn/JsonWriter.java

Example builder use
https://github.com/dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/exec/vector/complex/fn/JsonReader.java

On Fri, Aug 30, 2019, 8:14 PM Micah Kornfield <emkornfield@gmail.com> wrote:

> Hi Simon,
> A couple notes:
> - Scalars are a C++ thing, There are ValueHolders in Java but i'm not sure
> what you want.
> - Dremio has a JDBC adaptor [1] that might be worth looking at (or maybe
> porting pieces of it into Arrow).
>
> Thanks,
> Micah
>
> [1]
> https://github.com/dremio/dremio-oss/blob/04e0387d474f1408731da0029aef7ecfad5e4d08/client/jdbc/src/main/java/com/dremio/jdbc/impl/DremioResultSetImpl.java
>
>
>
> On Fri, Aug 30, 2019 at 12:03 PM Simon Dumke <simon.dumke@ipp.mpg.de>
> wrote:
>
>> Hi Ben,
>>
>> thanks for the suggestion, i'll loon into it!
>>
>> Regards,
>> Simon
>>
>> Am 30. August 2019 20:59:23 schrieb Ben Kietzman <
>> ben.kietzman@rstudio.com>:
>>
>>> Hi Simon,
>>>
>>> If you're interested in adding a record interface, the Scalar classes
>>> might be a good place to start. They represent a value from an array slot
>>> and it should be fairly straightforward to extract a table row as a
>>> StructScalar
>>>
>>> On Fri, Aug 30, 2019 at 1:27 PM Wes McKinney <wesmckinn@gmail.com>
>>> wrote:
>>>
>>>> hi Simon -- I don't think there is any such Row accessor class in Java
>>>> but you are welcome to contribute one to the project. For performance
>>>> sensitive applications, using a record interface might not be the best
>>>> idea, but I can understand the convenience for some uses cases.
>>>>
>>>> - Wes
>>>>
>>>> On Fri, Aug 30, 2019 at 4:55 AM Simon Dumke <simon.dumke@ipp.mpg.de>
>>>> wrote:
>>>> >
>>>> > Hi all,
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > I did not find anything (and so: no definite answer) in the docs, so
i
>>>> > thought to ask here:
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > Does Arrow (and at this point my main concern is Arrow for java)
>>>> support
>>>> > any type of concept that allows a "record level access" (so, a "row")
>>>> to
>>>> > data in an Arrow RecordBatch or Table? I would have thougt that even
>>>> in
>>>> > column-oriented analytics etc. this would be a common last step access
>>>> > pattern over many use cases, but i could not find any references to
>>>> such a
>>>> > thing.
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > Thanks and kind regards,
>>>> > Simon
>>>> >
>>>> >
>>>>
>>>
>>

Mime
View raw message