flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrea Spina <andrea.sp...@radicalbit.io>
Subject Providing Custom Serializer for Generic Type
Date Thu, 04 Jul 2019 06:37:19 GMT
Dear community,
in my job, I run with a custom event type *MyClass* which is a sort of
"generic event" that I handle all along my streaming flow both as an event
(DataStream[MyClass]) and as a managed state.

I see that Flink warns me about generic serialization of
*MyClass*
 INFO [run-main-0] (TypeExtractor.java:1818) - class io.radicalbit.MyClass
does not contain a setter for field io$radicalbit$MyClass$$schema
 INFO [run-main-0] (TypeExtractor.java:1857) - Class class
io.radicalbit.MyClass cannot be used as a POJO type because not all fields
are valid POJO fields, and must be processed as GenericType. Please read
the Flink documentation on "Data Types & Serialization" for details of the
effect on performance.
 INFO [run-main-0] (TypeExtractor.java:1818) - class io.radicalbit.MyClass
does not contain a setter for field io$radicalbit$MyClass$schema

So that I wanted to provide my custom serializer for MyClass, trying first
to register the Java one to check if the system recognizes it so I followed
[1] but it seems that it is not considered.

I read then about [2] (the case is way akin to mine) and AFAIU I need to
implement a custom TypeInformation and TypeSerializer for my class as
suggested in [3] because Flink will ignore my registered serializer as long
as it considers my type as *generic*.

config.registerTypeWithKryoSerializer(classOf[MyClass], classOf[RadicalSerde])


My question finally is: Do I need to provide this custom classes? Is there
any practical example for creating custom information like the above
mentioned? I have had a quick preliminary look at it but seems that I need
to provide a non-trivial amount of information to TypeInformation and
TypeSerializer interfaces.

Thank you for your excellent work and help.

Cheers.

[1] -
https://ci.apache.org/projects/flink/flink-docs-stable/dev/custom_serializers.html
[2] -
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Custom-Serializer-for-Avro-GenericRecord-td25433.html
[3] -
https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/types_serialization.html#defining-type-information-using-a-factory
-- 
*Andrea Spina*
Head of R&D @ Radicalbit Srl
Via Giovanni Battista Pirelli 11, 20124, Milano - IT

Mime
View raw message