flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrea Sella ...@alkagin.xyz>
Subject Re: Flink Serialization and case class fields limit
Date Fri, 16 Nov 2018 19:05:40 GMT
Hi Andrey,

My bad, I forgot to say that I am using Scala 2.11, that’s why I asked
about the limitation, and Flink 1.5.5.

If I recall correctly CaseClassSerilizer and CaseClassTypeInfo don’t rely
on unapply and tupled functions, so I'd say that Flink doesn't have this
kind of limitation with Scala 2.11. Correct?

Thank you,
Andrea
On Fri, 16 Nov 2018 at 19:34, Andrey Zagrebin <andrey@data-artisans.com>
wrote:

> Hi Andrea,
>
> 22 limit comes from Scala [1], not Flink.
> I am not sure about any repo for the post, but I also cc'ed Fabian, maybe
> he will point to some if it exists.
>
> Best,
> Andrey
>
> [1] https://underscore.io/blog/posts/2016/10/11/twenty-two.html
>
>
> On 16 Nov 2018, at 13:10, Andrea Sella <me@alkagin.xyz> wrote:
>
> Hey squirrels,
>
> I've started to study more in-depth Flink Serialization and its "type
> system".
>
> I have a generated case class using scalapb that has more than 30 fields;
> I've seen that Flink still uses the CaseClassSerializer, the
> TypeInformation is CaseClassTypeInfo, even if in the docs[1] is written
> differently (22 fields limit). I'd have expected a GenericTypeInfo, but all
> is well because the CaseClassSerializer is faster than Kryo. Did I
> misunderstand the documentation or don't the limitation apply anymore?
>
> Another thing, I've read "Juggling with Bits and Bytes"[2] blog post an I
> would like to replicate the experiment with some tailored changes to deep
> dive even better in the topic. Is the source code in Github or somewhere
> else?
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/types_serialization.html#flinks-typeinformation-class
> [2]
> https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.html
>
> Thank you,
> Andrea
>
>
>

Mime
View raw message