flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rinat <r.shari...@cleverdata.ru>
Subject In-Memory state serialization with kryo fails
Date Tue, 12 Feb 2019 18:56:27 GMT
Hi mates !

I’ve implemented a job, that stores it’s progress using MapState[K, V], where K - is java.lang.String,
and V - is a collection of some typed objects java.util.List[SomeClass[_]]
When Flink is trying to serialize this state, it is using kryo serializer for value object
and fails with StackOverflowException

	at java.util.HashMap.hash(HashMap.java:338)
	at java.util.HashMap.get(HashMap.java:556)
	at com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:43)
This problem is related with the known bug in kryo (https://github.com/EsotericSoftware/kryo/issues/341),
and reveals itself only when type of SomeClass is a java.util.BitSet. 

I’ve checked my job locally (from IDE) with latest (4.0.2 <https://mvnrepository.com/artifact/com.esotericsoftware/kryo/4.0.2>)
kryo lib, and it works fine, but I couldn’t change kryo version for distributed mode, because
it’s packaged into fat-jar (flink-dist_2.11-1.6.1.jar), that
contains all runtime dependencies for Flink.

Maybe you can give me any advices, how to solve this issue, or register a separate serializers
for this case ?

Thx for your help.

Sincerely yours,
Rinat Sharipov
Software Engineer at 1DMP CORE Team

email: r.sharipov@cleverdata.ru <mailto:a.totmakov@cleverdata.ru>
mobile: +7 (925) 416-37-26

make your data clever

View raw message