flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arkay <robkee...@gmail.com>
Subject Re: "Memory ran out" error when running connected components
Date Fri, 13 May 2016 12:28:24 GMT
Hi Vasia,

It seems to work OK up to about 50MB of input, and dies after that point. 
If i disable just this connected components step the rest of my program is
happy with the full 1.5GB test dataset.  It seems to be specifically limited
to GraphAlgorithms in my case.

Do you know what the units are when it is saying Partition memory: 8388608? 
If it is bytes then it sounds like its using around 256MB per hash table of
32 partitions (which is then multiplied by number of task slots i guess). 
Can this number be configured do you know?  Perhaps the windows version of
the JVM is defaulting it to a lower value than on Linux?


View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Memory-ran-out-error-when-running-connected-components-tp6888p6899.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

View raw message