Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
MIME-Version: 1.0
In-Reply-To: <56A0B242.4040005@dfki.de>
References: <56A0A947.1060406@dfki.de>
	<CAC27z=M5MeTwCqEnQFRYZ1m-ExxJ+XGzmBN=9cvR3oE+sMrwqA@mail.gmail.com>
	<56A0B242.4040005@dfki.de>
Date: Thu, 21 Jan 2016 12:05:56 +0100
Message-ID: 
 <CAC27z=Os_vguXvQqRMBSzKHup3s_HTat_xMQUKDtBVo-Mvg0XQ@mail.gmail.com>
Subject: Re: Stackoverflow in Serialization
From: Till Rohrmann <trohrmann@apache.org>
To: dev@flink.apache.org
Content-Type: multipart/alternative; boundary=089e011764313938ec0529d6168b

--089e011764313938ec0529d6168b
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

As you can see from the namespace and the object names, it's not our code
which causes the stack overflow. It's Kryo code which you're calling and
thus we have not control over it.

And in fact, I doubt that the problem is related to the number of elements
in your map but rather to the structure of the types. My guess is that you
have a type which contains a field of another type, which contains a field
of another type, and so on. You should check that.

Cheers,
Till

On Thu, Jan 21, 2016 at 11:26 AM, Hilmi Yildirim <Hilmi.Yildirim@dfki.de>
wrote:

> Hi,
> I have a complex object with a MutableList and a Map containing thousands
> of objects. Here is a snippet of the exception. The actual exception is
> much longer. There, you can see that the exception is caused by too many
> recursive calls. For each recursive call the stack size increases because
> the parameters of each resursion level are saved on the stack. I am not
> sure but I suggest that the Map is the problem.
>
> Exception in thread "main" java.lang.StackOverflowError
>     at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:445)
>     at
> com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultCla=
ssResolver.java:79)
>     at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:488)
>     at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:593)
>     at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:36=
)
>     at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33=
)
>     at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
>     at
> com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:=
61)
>     at
> com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializ=
er.java:495)
>     at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
>     at
> com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:=
61)
>     at
> com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializ=
er.java:495)
>     at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
>     at
> com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:=
61)
>     at
> com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializ=
er.java:495)
>     at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
>
>
>
> Best Regards,
> Hilmi
>
>
>
>
>
>
> Am 21.01.2016 um 11:14 schrieb Till Rohrmann:
>
>> Hi Hilmi,
>>
>> I've never seen such a problem. Could you please provide a bit more
>> background information. Which types are you serializing? Which serialize=
r
>> are you using? What do you mean with recursion? Maybe you can provide a
>> simple example program which reproduces the problem.
>>
>> The thing is that if you have a composite data type, then each field wil=
l
>> be serialized with it's own serializer. Thus, if you data type is strong=
ly
>> nested, then you will end up with a long chain of nested serializer call=
s.
>> In order to cause a stack overflow exception, you would have to have a
>> really deeply nested data structure. Maybe you can flatten it a bit.
>>
>> Cheers,
>> Till
>>
>> On Thu, Jan 21, 2016 at 10:47 AM, Hilmi Yildirim <Hilmi.Yildirim@dfki.de=
>
>> wrote:
>>
>> Hi,
>>> I am using flink to train a ML model. There, I need to broadcast huge
>>> broadcast variables in each iteration. During the serialization I get a
>>> Stackoverflow exception. If I increase the Stack size then it works wel=
l.
>>> The Stackoverflow could be avoided If you do not use recursion in
>>> serialization. You could use iterations instead, I know it is more
>>> complex
>>> but this would avoid a stack overflow.
>>>
>>> Best Regards,
>>> Hilmi
>>>
>>>
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> Hilmi Yildirim, M.Sc.
> Researcher
>
> DFKI GmbH
> Intelligente Analytik f=C3=BCr Massendaten
> DFKI Projektb=C3=BCro Berlin
> Alt-Moabit 91c
> D-10559 Berlin
> Phone: +49 30 23895 1814
>
> E-Mail: Hilmi.Yildirim@dfki.de <mailto:Hilmi.Yildirim@dfki.de>
>
> -------------------------------------------------------------
> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>
> Geschaeftsfuehrung:
> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
> Dr. Walter Olthoff
>
> Vorsitzender des Aufsichtsrats:
> Prof. Dr. h.c. Hans A. Aukes
>
> Amtsgericht Kaiserslautern, HRB 2313
> -------------------------------------------------------------
>
> --
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> Hilmi Yildirim, M.Sc.
> Researcher
>
> DFKI GmbH
> Intelligente Analytik f=C3=BCr Massendaten
> DFKI Projektb=C3=BCro Berlin
> Alt-Moabit 91c
> D-10559 Berlin
> Phone: +49 30 23895 1814
>
> E-Mail: Hilmi.Yildirim@dfki.de
>
> -------------------------------------------------------------
> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>
> Geschaeftsfuehrung:
> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
> Dr. Walter Olthoff
>
> Vorsitzender des Aufsichtsrats:
> Prof. Dr. h.c. Hans A. Aukes
>
> Amtsgericht Kaiserslautern, HRB 2313
> -------------------------------------------------------------
>
>

--089e011764313938ec0529d6168b--