ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Mashenkov <andrey.mashen...@gmail.com>
Subject Re: Ignite Client Heap out of Memory issue
Date Fri, 30 Mar 2018 17:36:07 GMT
Hi Shawn,

1. Ignite use off heap to store cache entries. Client store no cache data.
Cache in LOCAL mode can be used on client side, and it should use offheap
of course.

All data client retreive from server will be in offheap

2. It is not IgniteOutOfMemory error, but JVM OOM.
So, try to investigate if there is a memory leak in your code.

On Fri, Mar 30, 2018 at 6:36 AM, shawn.du <shawn.du@neulion.com.cn> wrote:

> Hi,
>
> My Ignite client heap OOM yesterday.  This is the first time we encounter
> this issue.
>
> My ignite client colocates within Storm worker process. this issue cause
> storm worker restart.
> I have several questions about it: our ignite version is 2.3.0
> 1) if ignite in client mode, it use offheap? how to set the max
> onheap/offheap memory to use.
> 2) our storm worker have 8G memory, ignite client print OOM, it doesn't
> trigger storm worker to dump the heap.
>     but we get a ignite server's heap dump.  ignite server didn't die.
> The ignite server's heap dump is very small. only have 200M.
>     which process is OOM? worker or ignite server?
>
> This is logs:  Thanks in advance.
>
>   Suppressed: org.apache.ignite.IgniteCheckedException: Failed to update
> keys on primary node.
>                 at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.UpdateErrors.
> addFailedKeys(UpdateErrors.java:124) ~[stormjar.jar:?]
>                 at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridNearAtomicUpdateResponse.
> addFailedKeys(GridNearAtomicUpdateResponse.java:342) ~[stormjar.jar:?]
>                 at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.
> updateAllAsyncInternal0(GridDhtAtomicCache.java:1784) ~[stormjar.jar:?]
>                 at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.
> updateAllAsyncInternal(GridDhtAtomicCache.java:1627) ~[stormjar.jar:?]
>                 at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.
> processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3054)
> ~[stormjar.jar:?]
>                 at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.
> access$400(GridDhtAtomicCache.java:129) ~[stormjar.jar:?]
>                 at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache$
> 5.apply(GridDhtAtomicCache.java:265) ~[stormjar.jar:?]
>                 at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache$
> 5.apply(GridDhtAtomicCache.java:260) ~[stormjar.jar:?]
>                 at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.processMessage(GridCacheIoManager.java:1060)
> ~[stormjar.jar:?]
>                 at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
> ~[stormjar.jar:?]
>                 at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
> ~[stormjar.jar:?]
>                 at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
> ~[stormjar.jar:?]
>                 at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.access$100(GridCacheIoManager.java:99)
> ~[stormjar.jar:?]
>                 at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
> ~[stormjar.jar:?]
>                 at org.apache.ignite.internal.managers.communication.
> GridIoManager.invokeListener(GridIoManager.java:1555) ~[stormjar.jar:?]
>                 at org.apache.ignite.internal.managers.communication.
> GridIoManager.processRegularMessage0(GridIoManager.java:1183)
> ~[stormjar.jar:?]
>                 at org.apache.ignite.internal.managers.communication.
> GridIoManager.access$4200(GridIoManager.java:126) ~[stormjar.jar:?]
>                 at org.apache.ignite.internal.managers.communication.
> GridIoManager$9.run(GridIoManager.java:1090) ~[stormjar.jar:?]
>                 at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
> ~[stormjar.jar:?]
>                 at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
>                 Suppressed: java.lang.OutOfMemoryError: Java heap space
>                         at org.apache.ignite.internal.processors.cache.
> IncompleteCacheObject.<init>(IncompleteCacheObject.java:44)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.cacheobject.IgniteCacheObjectProcessorImpl.toCacheObject(
> IgniteCacheObjectProcessorImpl.java:191) ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:404)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:248)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:174)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:62)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.query.h2.database.io.H2ExtrasInnerIO.
> getLookupRow(H2ExtrasInnerIO.java:114) ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.query.h2.database.io.H2ExtrasInnerIO.
> getLookupRow(H2ExtrasInnerIO.java:36) ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.query.h2.database.H2Tree.getRow(H2Tree.java:123)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.query.h2.database.H2Tree.getRow(H2Tree.java:40)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4372)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.query.h2.database.H2Tree.compare(H2Tree.java:200)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.query.h2.database.H2Tree.compare(H2Tree.java:40)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4359)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4279)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.cache.persistence.tree.BPlusTree.access$1500(BPlusTree.java:81)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:261)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4697)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4682)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:158)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.cache.persistence.DataStructure.read(DataStructure.java:319)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1823)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1842)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.cache.persistence.tree.BPlusTree.doRemove(BPlusTree.java:1752)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.cache.persistence.tree.BPlusTree.remove(BPlusTree.java:1578)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.query.h2.database.H2TreeIndex.remove(H2TreeIndex.java:216)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.query.h2.opt.GridH2Table.doUpdate(GridH2Table.java:504)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.query.h2.opt.GridH2Table.update(GridH2Table.java:423)
> ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.processors.query.h2.
> IgniteH2Indexing.remove(IgniteH2Indexing.java:580) ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.processors.query.
> GridQueryProcessor.remove(GridQueryProcessor.java:2334) ~[stormjar.jar:?]
>                         at org.apache.ignite.internal.
> processors.cache.query.GridCacheQueryManager.remove(
> GridCacheQueryManager.java:461)
>
>
>
>
>
> Thanks
> Shawn
>
>


-- 
Best regards,
Andrey V. Mashenkov

Mime
View raw message