ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vadim Dedkov <dedko...@gmail.com>
Subject Re: OutOfMemoryException
Date Fri, 17 Mar 2017 14:52:01 GMT
I wrote to Aleksey Shcherbakov and he advised me to set parameter
"GridDatabaseConfiguration.pageCacheSize = 53687091200" (50 GB).
After that I have changed my GG config like this (be careful, it was not
GridDatabaseConfiguration-bean exactly):

<beans xmlns="http://www.springframework.org/schema/beans" ... >
    <bean id="ignite.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
        <property name="clientMode" value="false"/>
        ...
        <property name="memoryConfiguration">
            <bean
class="org.apache.ignite.configuration.MemoryConfiguration">
                <property name="pageCacheSize" value="53687091200" />
            </bean>
        </property>
    </bean>
</beans>

This helped me to overcome OutOfMemoryException.


2017-02-17 6:04 GMT+03:00 Denis Magda <dmagda@gridgain.com>:

> Hello Vadim,
>
> This is a support channel of Apache Ignite community. You’ll not get
> support for GridGain here.
>
> Please talk to your colleagues or Alexey Scherbakov (cc-ed) asking on how
> get access to FreshDesk which is GridGain’s support portal.
>
>
>
> Regards,
> --
> Denis Magda
> Product Manager, GridGain Systems <https://www.gridgain.com>
>
> Begin forwarded message:
>
> *From: *Vadim Dedkov <dedkovva@gmail.com>
> *Subject: **OutOfMemoryException*
> *Date: *February 16, 2017 at 8:58:15 AM PST
> *To: *user@ignite.apache.org
> *Reply-To: *user@ignite.apache.org
>
> Hello to all.
> I use GG version 8.0.1.ea7 & ignite version 1.10.1.ea7.
> On my cluster I have 4 server nodes.
> Each node have 8 cpu and 16gb memory.
> GG visor info on each node: Heap memory initialized: 10gb, Heap memory
> maximum: 10gb.
> I tried to process file (~4.7gb) with Hadoop MR: parse and write file data
> to GG cache.
>
> And after a while since start job I got errors (end of letter).
> I tried to analise error log & source code and one of my versions is: GG
> tries to swap memory on hard disk --> I don't have enough free space in GG
> directory --> error.
> But I didn't setup cache config "swapEnabled" and I cannot find such
> config option in Ignite 1.10.ea7 source code.
> And in this hypothesis I cannot realise, why GG tries to use swap: 10gb *
> 4node is not enough for 4.7 gb data?
>
>
> On one of my GG node:
> [16:35:08,442][SEVERE][sys-#10%null%][GridDhtAtomicCache]
> <calcScorePointsCacheIn_2001> Unexpected exception during cache update
> class org.apache.ignite.internal.mem.OutOfMemoryException: null
>         at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.
> allocatePage(PageMemoryNoStoreImpl.java:242)
>         at org.apache.ignite.internal.processors.cache.database.
> freelist.FreeListImpl.allocateDataPage(FreeListImpl.java:295)
>         at org.apache.ignite.internal.processors.cache.database.
> freelist.FreeListImpl.insertDataRow(FreeListImpl.java:325)
>         at org.apache.ignite.internal.processors.cache.database.
> RowStore.addRow(RowStore.java:80)
>         at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.update(
> IgniteCacheOffheapManagerImpl.java:842)
>         at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffheapManagerImpl.update(IgniteCacheOffheapManagerImpl.
> java:302)
>         at org.apache.ignite.internal.processors.cache.
> GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3576)
>         at org.apache.ignite.internal.processors.cache.
> GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2093)
>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2353)
>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(
> GridDhtAtomicCache.java:1755)
>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(
> GridDhtAtomicCache.java:1593)
>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest
> (GridDhtAtomicCache.java:3059)
>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:131)
>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:288)
>         at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:283)
>         at org.apache.ignite.internal.processors.cache.GridCacheIoManager.
> processMessage(GridCacheIoManager.java:758)
>         at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.onMessage0(GridCacheIoManager.java:363)
>         at org.apache.ignite.internal.processors.cache.GridCacheIoManager.
> handleMessage(GridCacheIoManager.java:287)
>         at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.access$000(GridCacheIoManager.java:89)
>         at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager$1.onMessage(GridCacheIoManager.java:232)
>         at org.apache.ignite.internal.managers.communication.
> GridIoManager.invokeListener(GridIoManager.java:1369)
>         at org.apache.ignite.internal.managers.communication.
> GridIoManager.processRegularMessage0(GridIoManager.java:997)
>         at org.apache.ignite.internal.managers.communication.
> GridIoManager.access$2100(GridIoManager.java:113)
>         at org.apache.ignite.internal.managers.communication.
> GridIoManager$6.run(GridIoManager.java:950)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
>
> And in MR job log:
> Error: org.apache.ignite.cache.CachePartialUpdateException: Failed to
> update keys (retry update if possible).: [00-1-10060940]
>          at org.apache.ignite.internal.processors.cache.GridCacheUtils.
> convertToCacheException(GridCacheUtils.java:1478)
>          at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.
> cacheException(IgniteCacheProxy.java:2118)
>          at org.apache.ignite.internal.processors.cache.
> IgniteCacheProxy.put(IgniteCacheProxy.java:1315)
>          at ru.sberbank.bigdata.mlfw.hadoop.hadoop_to_gridgain_table_load.
> GridGainLineCacheFiller.fillCache(GridGainLineCacheFiller.scala:10)
>          at ru.sberbank.bigdata.mlfw.hadoop.hadoop_to_gridgain_table_load.
> ValueGroupingReducer.reduce(ValueGroupingReducer.scala:36)
>          at ru.sberbank.bigdata.mlfw.hadoop.hadoop_to_gridgain_table_load.
> ValueGroupingReducer.reduce(ValueGroupingReducer.scala:16)
>          at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
>          at org.apache.hadoop.mapred.ReduceTask.runNewReducer(
> ReduceTask.java:627)
>          at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>          at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>          at java.security.AccessController.doPrivileged(Native Method)
>          at javax.security.auth.Subject.doAs(Subject.java:422)
>          at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1709)
>          at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: class org.apache.ignite.internal.processors.cache.
> CachePartialUpdateCheckedException: Failed to update keys (retry update
> if possible).: [00-1-10060940]
>          at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridNearAtomicSingleUpdateFuture.onResult(
> GridNearAtomicSingleUpdateFuture.java:227)
>          at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRespons
> e(GridDhtAtomicCache.java:3077)
>          at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.access$700(GridDhtAtomicCache.java:131)
>          at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache$6.apply(GridDhtAtomicCache.java:306)
>          at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache$6.apply(GridDhtAtomicCache.java:301)
>          at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.processMessage(GridCacheIoManager.java:758)
>          at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.onMessage0(GridCacheIoManager.java:363)
>          at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.handleMessage(GridCacheIoManager.java:287)
>          at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.access$000(GridCacheIoManager.java:89)
>          at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager$1.onMessage(GridCacheIoManager.java:232)
>          at org.apache.ignite.internal.managers.communication.
> GridIoManager.invokeListener(GridIoManager.java:1369)
>          at org.apache.ignite.internal.managers.communication.
> GridIoManager.processRegularMessage0(GridIoManager.java:997)
>          at org.apache.ignite.internal.managers.communication.
> GridIoManager.access$2100(GridIoManager.java:113)
>          at org.apache.ignite.internal.managers.communication.
> GridIoManager$6.run(GridIoManager.java:950)
>          at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>          at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>          Suppressed: class org.apache.ignite.IgniteCheckedException:
> Failed to update keys on primary node.
>                  at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridNearAtomicUpdateResponse.
> addFailedKeys(GridNearAtomicUpdateResponse.java:370)
>                  at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.
> updateAllAsyncInternal0(GridDhtAtomicCache.java:1828)
>                  at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.
> updateAllAsyncInternal(GridDhtAtomicCache.java:1593)
>                  at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.
> processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3059)
>                  at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.
> access$600(GridDhtAtomicCache.java:131)
>                  at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache$
> 5.apply(GridDhtAtomicCache.java:288)
>                  at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache$
> 5.apply(GridDhtAtomicCache.java:283)
>                  ... 12 more
>                  Suppressed: class org.apache.ignite.internal.mem.OutOfMemoryException:
> null
>                           at org.apache.ignite.internal.pagemem.impl.
> PageMemoryNoStoreImpl.allocatePage(PageMemoryNoStoreImpl.java:242)
>                           at org.apache.ignite.internal.
> processors.cache.database.freelist.FreeListImpl.
> allocateDataPage(FreeListImpl.java:295)
>                           at org.apache.ignite.internal.
> processors.cache.database.freelist.FreeListImpl.
> insertDataRow(FreeListImpl.java:325)
>                           at org.apache.ignite.internal.
> processors.cache.database.RowStore.addRow(RowStore.java:80)
>                           at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.update(
> IgniteCacheOffheapManagerImpl.java:842)
>                           at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffheapManagerImpl.update(IgniteCacheOffheapManagerImpl.
> java:302)
>                           at org.apache.ignite.internal.processors.cache.
> GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3576)
>                           at org.apache.ignite.internal.processors.cache.
> GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2093)
>                           at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(
> GridDhtAtomicCache.java:2353)
>                           at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.
> updateAllAsyncInternal0(GridDhtAtomicCache.java:1755)
>                           ... 17 more
>
>
>
> --
> _______________             _______________
> Best regards,                    С уважением
> Vadim Dedkov.                  Вадим Дедков.
>
>
>


-- 
_______________             _______________
Best regards,                    С уважением
Vadim Dedkov.                  Вадим Дедков.

Mime
View raw message