flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ken Krugler <kkrugler_li...@transpac.com>
Subject RE: Kyro Intermittent Exception for Large Data
Date Thu, 18 Feb 2016 23:39:02 GMT
I've seen this type of error when using Kryo with a Cascading scheme I'd created.

In my case it happened when serializing a large object graph, where some of the classes didn't
have no-arg constructors.

The general fix was to set an instantiator strategy for Kryo - see:

https://github.com/ScaleUnlimited/cascading.utils/blob/master/src/main/java/com/scaleunlimited/cascading/local/KryoContext.java#L81

Don't know if Flink always sets up to use this (explicitly, or fallback) everywhere.

-- Ken

> From: Welly Tambunan
> Sent: February 17, 2016 6:10:03pm PST
> To: user@flink.apache.org
> Subject: Kyro Intermittent Exception for Large Data
> 
> Hi All, 
> 
> 
> We create a downsampling job that will respond to a request from RabbitMQ. However we
face different kind of exception intermittently when transferring large data. Is there any
setting that we can tune to make Kyro able to serialize and deserialize the large data ? 
> 
> I can't find any reference in the documentation about Kyro or Default Java serialization
in the latest documentation. Can someone show me the right place to look at ? As i think before
fink use Java serialization ?
> 
> Thanks a lot. 
> 
> ERROR 1
> 
> <image.png>
> 
> ERROR 2
> 
> <image.png>
> 
> 
> -- 
> Welly Tambunan
> Triplelands 
> 
> http://weltam.wordpress.com
> http://www.triplelands.com




--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr






Mime
View raw message