flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Debug OutOfMemory
Date Thu, 08 Oct 2015 16:23:49 GMT
Can you paste the exception stack trace?

On Thu, Oct 8, 2015 at 6:15 PM, KOSTIANTYN Kudriavtsev <
kudryavtsev.konstantin@gmail.com> wrote:

> It's DataSet program that performs simple filtering, crossjoin and
> aggregation.
>
> I'm using Hadoop S3 FileSystem (not Emr) as far as Flink's s3 connector
> doesn't work at all.
>
> Currently I have 3 taskmanagers each 5k MB, but I tried different
> configurations and all leads to the same exception
>
> *Sent from my ZenFone
> On Oct 8, 2015 12:05 PM, "Stephan Ewen" <sewen@apache.org> wrote:
>
>> Can you give us a bit more background?  What exactly is your program
>> doing?
>>
>>   - Are you running a DataSet program, or a DataStream program?
>>   - Is it one simple source that reads from S3, or are there multiple
>> sources?
>>   - What operations do you apply on the CSV file?
>>   - Are you using Flink's S3 connector, or the Hadoop S3 file system?
>>
>> Greetings,
>> Stephan
>>
>>
>> On Thu, Oct 8, 2015 at 5:58 PM, KOSTIANTYN Kudriavtsev <
>> kudryavtsev.konstantin@gmail.com> wrote:
>>
>>> Hi guys,
>>>
>>> I'm running FLink on EMR with 2 m3.xlarge (each 16 GB RAM) and trying
>>> to process 3.8 GB CSV data from S3. I'm surprised the fact that Flink
>>> failed with OutOfMemory: Java Heap space
>>>
>>> I tried to find the reason:
>>> 1) to identify TaskManager with a command ps aux | grep TaskManager
>>> 2) then build Heap histo:
>>> $ jmap -histo:live 19648 | head -n23
>>>  num     #instances         #bytes  class name
>>> ----------------------------------------------
>>>    1:        131018     3763501304  [B
>>>    2:         61022        7820352  <methodKlass>
>>>    3:         61022        7688456  <constMethodKlass>
>>>    4:          4971        5454408  <constantPoolKlass>
>>>    5:          4966        4582232  <instanceKlassKlass>
>>>    6:          4169        3003104  <constantPoolCacheKlass>
>>>    7:         15696        1447168  [C
>>>    8:          1291         638824  [Ljava.lang.Object;
>>>    9:          5318         506000  java.lang.Class
>>>
>>>
>>> Do you have any ideas what can be the reason and how it can be fixed?
>>> Is Flink uses out-of-heap memory?
>>>
>>>
>>> Thank you,
>>> Konstantin Kudryavtsev
>>>
>>
>>

Mime
View raw message