flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Debug OutOfMemory
Date Thu, 08 Oct 2015 16:06:53 GMT
Hi Konstantin,

Flink uses managed memory only for its internal processing (sorting, hash
tables, etc.).
If you allocate too much memory in your user code, it can still fail with
an OOME. This can also happen for large broadcast sets.
Can you check how much much memory the JVM allocated and how much was
allocated as managed memory. You'll find that information in the task
manager log file.

Cheers,
Fabian

2015-10-08 17:58 GMT+02:00 KOSTIANTYN Kudriavtsev <
kudryavtsev.konstantin@gmail.com>:

> Hi guys,
>
> I'm running FLink on EMR with 2 m3.xlarge (each 16 GB RAM) and trying to
> process 3.8 GB CSV data from S3. I'm surprised the fact that Flink failed
> with OutOfMemory: Java Heap space
>
> I tried to find the reason:
> 1) to identify TaskManager with a command ps aux | grep TaskManager
> 2) then build Heap histo:
> $ jmap -histo:live 19648 | head -n23
>  num     #instances         #bytes  class name
> ----------------------------------------------
>    1:        131018     3763501304  [B
>    2:         61022        7820352  <methodKlass>
>    3:         61022        7688456  <constMethodKlass>
>    4:          4971        5454408  <constantPoolKlass>
>    5:          4966        4582232  <instanceKlassKlass>
>    6:          4169        3003104  <constantPoolCacheKlass>
>    7:         15696        1447168  [C
>    8:          1291         638824  [Ljava.lang.Object;
>    9:          5318         506000  java.lang.Class
>
>
> Do you have any ideas what can be the reason and how it can be fixed?
> Is Flink uses out-of-heap memory?
>
>
> Thank you,
> Konstantin Kudryavtsev
>

Mime
View raw message