flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <...@apache.org>
Subject Re: Crash in a simple "mapper style" streaming app likely due to a memory leak ?
Date Fri, 13 Nov 2015 14:59:11 GMT

> On 13 Nov 2015, at 15:49, LINZ, Arnaud <ALINZ@bouyguestelecom.fr> wrote:
> 
> Hi Robert,
>  
> Thanks, it works with 50% -- at least way past the previous crash point.
>  
> In my opinion (I lack real metrics), the part that uses the most memory is the M2 mapper,
instantiated once per slot.
> The most complex part is the Sink (it does use a lot of hdfs files, flushing threads
etc.) ; but I expect the “RichSinkFunction” to be instantiated only once per slot ? I’m
really surprised by that memory usage, I will try using a monitoring app on the yarn jvm to
understand.

In general it’s instantiated once per subtask. For your current deployment, it is one per
slot.

– Ufuk


Mime
View raw message