ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <zhangshuai.u...@gmail.com>
Subject RE: OOM when using Ignite as HDFS Cache
Date Fri, 14 Apr 2017 06:12:37 GMT
I think it's a kind of misconfiguration. The Ignite document just mentioned about how to configuration
HDFS as a secondary filesystem but nothing about how to restrict the memory usage to avoid
OOME. https://apacheignite.readme.io/v1.0/docs/igfs-secondary-file-system

Assume I configured the max JVM heap size to 1GB.
1. What would happen if I write very fast before Ignite write data to HDFS asynchronized?
2. What would happen if I want to write a 2GB file block to Ignite?

I expected:
1. Ignite would slow down the write performance to avoid OOME.
2. Ignite would break the 2GB file block into 512MB blocks & write them to HDFS to avoid
OOME.

Do we have configurations against above behaviors? I dig some items from source code &
Ignite Web Console, but seems they are not working fine. 

<property name="fragmentizerConcurrentFiles" value="3"/>
<property name="dualModeMaxPendingPutsSize" value="10"/>
<property name="blockSize" value="536870912"/>
<property name="streamBufferSize" value="131072"/>
<property name="maxSpaceSize" value="6442450944"/>
<property name="maximumTaskRangeLength" value="536870912"/>
<property name="prefetchBlocks" value="2"/>
<property name="sequentialReadsBeforePrefetch" value="5"/>
<property name="defaultMode" value="DUAL_ASYNC" />

I also notice that Ignite write through file block size is set to 64MB. I mean I write a file
to Ignite with block size to 4GB, but I finally found it on HDFS with block size 64MB. Is
there any configuration for it?

-----Original Message-----
From: dkarachentsev [mailto:dkarachentsev@gridgain.com] 
Sent: Thursday, April 13, 2017 11:21 PM
To: user@ignite.apache.org
Subject: Re: OOM when using Ignite as HDFS Cache

Hi Shuai,

Could you please take heap dump on OOME and find what objects consume memory? There would
be a lot of byte[] objects, please find the nearest GC root for them.

Thanks!

-Dmitry.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/OOM-when-using-Ignite-as-HDFS-Cache-tp11900p11956.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.


Mime
View raw message