flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hilmi Yildirim <Hilmi.Yildi...@dfki.de>
Subject Re: Stackoverflow in Serialization
Date Thu, 21 Jan 2016 10:26:10 GMT
Hi,
I have a complex object with a MutableList and a Map containing 
thousands of objects. Here is a snippet of the exception. The actual 
exception is much longer. There, you can see that the exception is 
caused by too many recursive calls. For each recursive call the stack 
size increases because the parameters of each resursion level are saved 
on the stack. I am not sure but I suggest that the Map is the problem.

Exception in thread "main" java.lang.StackOverflowError
     at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:445)
     at 
com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
     at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:488)
     at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:593)
     at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:36)
     at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33)
     at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
     at 
com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
     at 
com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
     at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
     at 
com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
     at 
com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
     at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
     at 
com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
     at 
com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
     at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)



Best Regards,
Hilmi





Am 21.01.2016 um 11:14 schrieb Till Rohrmann:
> Hi Hilmi,
>
> I've never seen such a problem. Could you please provide a bit more
> background information. Which types are you serializing? Which serializer
> are you using? What do you mean with recursion? Maybe you can provide a
> simple example program which reproduces the problem.
>
> The thing is that if you have a composite data type, then each field will
> be serialized with it's own serializer. Thus, if you data type is strongly
> nested, then you will end up with a long chain of nested serializer calls.
> In order to cause a stack overflow exception, you would have to have a
> really deeply nested data structure. Maybe you can flatten it a bit.
>
> Cheers,
> Till
>
> On Thu, Jan 21, 2016 at 10:47 AM, Hilmi Yildirim <Hilmi.Yildirim@dfki.de>
> wrote:
>
>> Hi,
>> I am using flink to train a ML model. There, I need to broadcast huge
>> broadcast variables in each iteration. During the serialization I get a
>> Stackoverflow exception. If I increase the Stack size then it works well.
>> The Stackoverflow could be avoided If you do not use recursion in
>> serialization. You could use iterations instead, I know it is more complex
>> but this would avoid a stack overflow.
>>
>> Best Regards,
>> Hilmi
>>

==================================================================
Hilmi Yildirim, M.Sc.
Researcher

DFKI GmbH
Intelligente Analytik für Massendaten
DFKI Projektbüro Berlin
Alt-Moabit 91c
D-10559 Berlin
Phone: +49 30 23895 1814

E-Mail: Hilmi.Yildirim@dfki.de <mailto:Hilmi.Yildirim@dfki.de>

-------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern

Geschaeftsfuehrung:
Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
Dr. Walter Olthoff

Vorsitzender des Aufsichtsrats:
Prof. Dr. h.c. Hans A. Aukes

Amtsgericht Kaiserslautern, HRB 2313
-------------------------------------------------------------

-- 
==================================================================
Hilmi Yildirim, M.Sc.
Researcher

DFKI GmbH
Intelligente Analytik für Massendaten
DFKI Projektbüro Berlin
Alt-Moabit 91c
D-10559 Berlin
Phone: +49 30 23895 1814

E-Mail: Hilmi.Yildirim@dfki.de

-------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern

Geschaeftsfuehrung:
Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
Dr. Walter Olthoff

Vorsitzender des Aufsichtsrats:
Prof. Dr. h.c. Hans A. Aukes

Amtsgericht Kaiserslautern, HRB 2313
-------------------------------------------------------------


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message