flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Till Rohrmann <trohrm...@apache.org>
Subject Re: Stackoverflow in Serialization
Date Thu, 21 Jan 2016 10:14:18 GMT
Hi Hilmi,

I've never seen such a problem. Could you please provide a bit more
background information. Which types are you serializing? Which serializer
are you using? What do you mean with recursion? Maybe you can provide a
simple example program which reproduces the problem.

The thing is that if you have a composite data type, then each field will
be serialized with it's own serializer. Thus, if you data type is strongly
nested, then you will end up with a long chain of nested serializer calls.
In order to cause a stack overflow exception, you would have to have a
really deeply nested data structure. Maybe you can flatten it a bit.

Cheers,
Till

On Thu, Jan 21, 2016 at 10:47 AM, Hilmi Yildirim <Hilmi.Yildirim@dfki.de>
wrote:

> Hi,
> I am using flink to train a ML model. There, I need to broadcast huge
> broadcast variables in each iteration. During the serialization I get a
> Stackoverflow exception. If I increase the Stack size then it works well.
> The Stackoverflow could be avoided If you do not use recursion in
> serialization. You could use iterations instead, I know it is more complex
> but this would avoid a stack overflow.
>
> Best Regards,
> Hilmi
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message