spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koert Kuipers <ko...@tresata.com>
Subject Re: encoders for more complex types
Date Thu, 27 Oct 2016 20:05:34 GMT
well i was using Aggregators that returned sequences of structs, or structs
with sequences inside etc. and got compilation errors on the codegen.

i didnt bother trying to reproduce it so far, or post it, since what i did
was beyond the supposed usage anyhow.


On Thu, Oct 27, 2016 at 4:02 PM, Herman van Hövell tot Westerflier <
hvanhovell@databricks.com> wrote:

> What kind of difficulties are you experiencing?
>
> On Thu, Oct 27, 2016 at 9:57 PM, Koert Kuipers <koert@tresata.com> wrote:
>
>> i have been pushing my luck a bit and started using ExpressionEncoder for
>> more complex types like sequences of case classes etc. (where the case
>> classes only had primitives and Strings).
>>
>> it all seems to work but i think the wheels come off in certain cases in
>> the code generation. i guess this is not unexpected, after all what i am
>> doing is not yet supported.
>>
>> is there a planned path forward to support more complex types with
>> encoders? it would be nice if we can at least support all types that
>> spark-sql supports in general for DataFrame.
>>
>> best, koert
>>
>
>

Mime
View raw message