ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "shawn.du" <shawn...@neulion.com.cn>
Subject Ignite Client Heap out of Memory issue
Date Fri, 30 Mar 2018 03:36:42 GMT
<html>
<head>
    <meta http-equiv='Content-Type' content='text/html; charset=UTF-8'>
</head>
<body>
<style>
    font{
        line-height: 1.7;
    }
</style>
<div style = 'font-family:"微软雅黑"; font-size: 14px; color:#000000; line-height:1.7;'>
    
<div><span>Hi,</span></div><div><span><br></span></div><div>My
Ignite client heap OOM yesterday. &nbsp;This is the first time we encounter this issue.</div><div><br></div><div>My
ignite client colocates within Storm worker process. this issue cause storm worker restart.</div><div>I
have several questions about it: our ignite version is 2.3.0</div><div>1) if ignite
in client mode, it use offheap? how to set the max onheap/offheap memory to use.</div><div>2)
our storm worker have 8G memory, ignite client print OOM, it doesn't trigger storm worker
to dump the heap.</div><div>&nbsp; &nbsp; but we get a ignite server's
heap dump. &nbsp;ignite server didn't die. &nbsp;The ignite server's heap dump is
very small. only have 200M.</div><div>&nbsp; &nbsp; which process is OOM?
worker or ignite server?</div><div><br></div><div>This is logs:
&nbsp;Thanks in advance.</div><div><br></div><div><div>&nbsp;
Suppressed: org.apache.ignite.IgniteCheckedException: Failed to update keys on primary node.</div><div>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.UpdateErrors.addFailedKeys(UpdateErrors.java:124)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateResponse.addFailedKeys(GridNearAtomicUpdateResponse.java:342)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1784)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1627)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3054)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:129)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:265)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:260)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1060)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]</div><div>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Suppressed: java.lang.OutOfMemoryError:
Java heap space</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.IncompleteCacheObject.&lt;init&gt;(IncompleteCacheObject.java:44)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cacheobject.IgniteCacheObjectProcessorImpl.toCacheObject(IgniteCacheObjectProcessorImpl.java:191)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:404)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:248)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:174)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:62)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.query.h2.database.io.H2ExtrasInnerIO.getLookupRow(H2ExtrasInnerIO.java:114)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.query.h2.database.io.H2ExtrasInnerIO.getLookupRow(H2ExtrasInnerIO.java:36)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:123)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:40)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4372)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:200)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:40)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4359)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4279)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1500(BPlusTree.java:81)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:261)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4697)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4682)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:158)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:319)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1823)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1842)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doRemove(BPlusTree.java:1752)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.remove(BPlusTree.java:1578)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.remove(H2TreeIndex.java:216)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.doUpdate(GridH2Table.java:504)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:423)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:580)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java:2334)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQueryManager.java:461)</div></div><div><br></div><div><span><br></span></div><div><span><br></span></div>
<div><span><br></span></div>
<div><span><br></span></div>
<div id="ntes-pcmail-signature" style="font-family:'微软雅黑'">
    <style type="text/css">
        a#ntes-pcmail-signature-default:hover {
            text-decoration: underline;
            color: #199cff;
            cursor: pointer;
        }
        a#ntes-pcmail-signature-default:active {
            text-decoration: underline;
            color: #246fce;
            cursor: pointer;
        }
    </style>

                <div style="font-size:14px; padding: 0;  margin:0;">
                    <div style="font-family:&quot;微软雅黑&quot;; font-size:
13px; color:#000000">
    <style>
        font{
            line-height: 1.5;
        }
    </style>
<div id="ntes-pcmail-signature-default" style="font-size:14px; color:#000; text-decoration:
none;">Thanks</div><div id="ntes-pcmail-signature-default" style="font-size:14px;
color:#000; text-decoration: none;">Shawn</div>
</div>
                </div>

</div><br>
<!--😀-->
</div>
</body>
</html>
Mime
View raw message