arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Micah Kornfield <emkornfi...@gmail.com>
Subject Re: Reccord-Level Access
Date Sat, 31 Aug 2019 15:21:13 GMT
That is a good point, I didn't initially think of these because they are
missing an adapter or documentation (creating a top level struct) on how to
make them work with a VectorSchemaRoot.

On Saturday, August 31, 2019, Jacques Nadeau <jacques@apache.org> wrote:

> I'm arrow Java there is a record level accesor facade called FieldReader.
> There is also a record level builder called ComplexWriter. Both allow
> arbitrary complex data to be worked with using method invocations. Won't be
> as efficient as a columnar algorithm but definitely much easier to get
> started. We use a heavily overloaded interface pattern to basically support
> dynamic typing.
>
> Example reader use
> https://github.com/dremio/dremio-oss/blob/master/sabot/
> kernel/src/main/java/com/dremio/exec/vector/complex/fn/JsonWriter.java
>
> Example builder use
> https://github.com/dremio/dremio-oss/blob/master/sabot/
> kernel/src/main/java/com/dremio/exec/vector/complex/fn/JsonReader.java
>
> On Fri, Aug 30, 2019, 8:14 PM Micah Kornfield <emkornfield@gmail.com>
> wrote:
>
>> Hi Simon,
>> A couple notes:
>> - Scalars are a C++ thing, There are ValueHolders in Java but i'm not
>> sure what you want.
>> - Dremio has a JDBC adaptor [1] that might be worth looking at (or maybe
>> porting pieces of it into Arrow).
>>
>> Thanks,
>> Micah
>>
>> [1] https://github.com/dremio/dremio-oss/blob/
>> 04e0387d474f1408731da0029aef7ecfad5e4d08/client/jdbc/src/
>> main/java/com/dremio/jdbc/impl/DremioResultSetImpl.java
>>
>>
>>
>> On Fri, Aug 30, 2019 at 12:03 PM Simon Dumke <simon.dumke@ipp.mpg.de>
>> wrote:
>>
>>> Hi Ben,
>>>
>>> thanks for the suggestion, i'll loon into it!
>>>
>>> Regards,
>>> Simon
>>>
>>> Am 30. August 2019 20:59:23 schrieb Ben Kietzman <
>>> ben.kietzman@rstudio.com>:
>>>
>>>> Hi Simon,
>>>>
>>>> If you're interested in adding a record interface, the Scalar classes
>>>> might be a good place to start. They represent a value from an array slot
>>>> and it should be fairly straightforward to extract a table row as a
>>>> StructScalar
>>>>
>>>> On Fri, Aug 30, 2019 at 1:27 PM Wes McKinney <wesmckinn@gmail.com>
>>>> wrote:
>>>>
>>>>> hi Simon -- I don't think there is any such Row accessor class in Java
>>>>> but you are welcome to contribute one to the project. For performance
>>>>> sensitive applications, using a record interface might not be the best
>>>>> idea, but I can understand the convenience for some uses cases.
>>>>>
>>>>> - Wes
>>>>>
>>>>> On Fri, Aug 30, 2019 at 4:55 AM Simon Dumke <simon.dumke@ipp.mpg.de>
>>>>> wrote:
>>>>> >
>>>>> > Hi all,
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > I did not find anything (and so: no definite answer) in the docs,
so
>>>>> i
>>>>> > thought to ask here:
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > Does Arrow (and at this point my main concern is Arrow for java)
>>>>> support
>>>>> > any type of concept that allows a "record level access" (so, a
>>>>> "row") to
>>>>> > data in an Arrow RecordBatch or Table? I would have thougt that
even
>>>>> in
>>>>> > column-oriented analytics etc. this would be a common last step
>>>>> access
>>>>> > pattern over many use cases, but i could not find any references
to
>>>>> such a
>>>>> > thing.
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > Thanks and kind regards,
>>>>> > Simon
>>>>> >
>>>>> >
>>>>>
>>>>
>>>

Mime
View raw message