ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "shawn.du" <shawn...@neulion.com.cn>
Subject Re: Ignite Client Heap out of Memory issue
Date Mon, 02 Apr 2018 05:22:35 GMT
<html>
<head>
    <meta http-equiv='Content-Type' content='text/html; charset=UTF-8'>
</head>
<body>
<style>
    font{
        line-height: 1.7;
    }
</style>
<div style = 'font-family:"微软雅黑"; font-size: 14px; color:#000000; line-height:1.7;'>
    <div>
<div><span>Hi Andrey,</span></div><div><br></div><div>Thanks
for your replay. It still confused me for:</div><div>1) for storm worker process,
If it is &nbsp;because of OOM and crashed. it should dump the heap. for I set &nbsp;<span
class="pun" style="font-family: inherit; font-style: inherit; font-variant: inherit; font-weight:
inherit; line-height: inherit; white-space: inherit; font-size: inherit; orphans: 2; widows:
2; border: 0px; font-stretch: inherit; vertical-align: baseline; box-sizing: inherit; color:
rgb(48, 51, 54); background-color: rgb(239, 240, 241);">-</span><span class="pln"
style="font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit;
line-height: inherit; white-space: inherit; font-size: inherit; orphans: 2; widows: 2; border:
0px; font-stretch: inherit; vertical-align: baseline; box-sizing: inherit; color: rgb(48,
51, 54); background-color: rgb(239, 240, 241);">XX</span><span class="pun" style="font-family:
inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit;
white-space: inherit; font-size: inherit; orphans: 2; widows: 2; border: 0px; font-stretch:
inherit; vertical-align: baseline; box-sizing: inherit; color: rgb(48, 51, 54); background-color:
rgb(239, 240, 241);">:+</span><span class="typ" style="font-family: inherit; font-style:
inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; white-space: inherit;
font-size: inherit; orphans: 2; widows: 2; border: 0px; font-stretch: inherit; vertical-align:
baseline; box-sizing: inherit; color: rgb(43, 145, 175); background-color: rgb(239, 240, 241);">HeapDumpOnOutOfMemoryError</span></div><div><span>&nbsp;
&nbsp; but it didn't. &nbsp;For storm worker, it behaves like a normal fatal error
which make storm worker restart.</span></div>
<div><span>2) It did make ignite server heap dump. for I analyzed the dump, it
is ignite server's. (seeing from PID and java command etc.)&nbsp;</span></div><div><span><br></span></div><div>anyone
can explain? Thanks.</div><div><span><br></span></div>
<div id="ntes-pcmail-signature" style="font-family:'微软雅黑'">
    <style type="text/css">
        a#ntes-pcmail-signature-default:hover {
            text-decoration: underline;
            color: #199cff;
            cursor: pointer;
        }
        a#ntes-pcmail-signature-default:active {
            text-decoration: underline;
            color: #246fce;
            cursor: pointer;
        }
    </style>

                <div style="font-size:14px; padding: 0;  margin:0;">
                    <div style="font-family:&quot;微软雅黑&quot;; font-size:
13px; color:#000000">
    <style>
        font{
            line-height: 1.5;
        }
    </style>
<div id="ntes-pcmail-signature-default" style="font-size:14px; color:#000; text-decoration:
none;">Thanks</div><div id="ntes-pcmail-signature-default" style="font-size:14px;
color:#000; text-decoration: none;">Shawn</div>
</div>
                </div>

</div><br>
</div><div class="J-reply" style="background-color:#f2f2f2;color:black;padding-top:6px;padding-bottom:6px;border-radius:3px;-moz-border-radius:3px;-webkit-border-radius:3px;margin-top:45px;margin-bottom:20px;font-family:'微软雅黑';">
    <div style="font-size:14px;line-height:1.5;word-break:break-all;margin-left:10px;margin-right:10px">On
<span class="mail-date">3/31/2018 01:36</span>,<a class="mail-to" style="text-decoration:none;color:#2a83f2;"
href="mailto:andrey.mashenkov@gmail.com">Andrey Mashenkov&lt;andrey.mashenkov@gmail.com&gt;</a>
wrote: </div>
</div>
<blockquote id="ntes-pcmail-quote" style="margin: 0; padding: 0; font-size: 14px; font-family:
'微软雅黑';">
<div dir="ltr">Hi Shawn,<div><br></div><div>1. Ignite use off
heap to store cache entries. Client store no cache data. <br>Cache in LOCAL mode can
be used on client side, and it should use offheap of course.</div><div><br></div><div>All
data client retreive from server will be in offheap</div><div><br></div><div>2.
It is not IgniteOutOfMemory error, but JVM OOM.&nbsp;</div><div>So, try to
investigate if there is a memory leak in your code.</div></div><div class="gmail_extra"><br><div
class="gmail_quote">On Fri, Mar 30, 2018 at 6:36 AM, shawn.du <span dir="ltr">&lt;<a
href="mailto:shawn.du@neulion.com.cn" target="_blank">shawn.du@neulion.com.cn</a>&gt;</span>
wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">

    

<div>

<div style="font-family:&quot;\005fae\008f6f\0096c5\009ed1&quot;;font-size:14px;color:#000000;line-height:1.7">
    
<div><span>Hi,</span></div><div><span><br></span></div><div>My
Ignite client heap OOM yesterday.&nbsp; This is the first time we encounter this issue.</div><div><br></div><div>My
ignite client colocates within Storm worker process. this issue cause storm worker restart.</div><div>I
have several questions about it: our ignite version is 2.3.0</div><div>1) if ignite
in client mode, it use offheap? how to set the max onheap/offheap memory to use.</div><div>2)
our storm worker have 8G memory, ignite client print OOM, it doesn't trigger storm worker
to dump the heap.</div><div>&nbsp; &nbsp; but we get a ignite server's
heap dump. &nbsp;ignite server didn't die.&nbsp; The ignite server's heap dump is
very small. only have 200M.</div><div>&nbsp; &nbsp; which process is OOM?
worker or ignite server?</div><div><br></div><div>This is logs:
&nbsp;Thanks in advance.</div><div><br></div><div><div>&nbsp;
Suppressed: org.apache.ignite.<wbr>IgniteCheckedException: Failed to update keys on
primary node.</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.distributed.<wbr>dht.atomic.UpdateErrors.<wbr>addFailedKeys(UpdateErrors.<wbr>java:124)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.distributed.<wbr>dht.atomic.<wbr>GridNearAtomicUpdateResponse.<wbr>addFailedKeys(<wbr>GridNearAtomicUpdateResponse.<wbr>java:342)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.distributed.<wbr>dht.atomic.GridDhtAtomicCache.<wbr>updateAllAsyncInternal0(<wbr>GridDhtAtomicCache.java:1784)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.distributed.<wbr>dht.atomic.GridDhtAtomicCache.<wbr>updateAllAsyncInternal(<wbr>GridDhtAtomicCache.java:1627)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.distributed.<wbr>dht.atomic.GridDhtAtomicCache.<wbr>processNearAtomicUpdateRequest<wbr>(GridDhtAtomicCache.java:3054)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.distributed.<wbr>dht.atomic.GridDhtAtomicCache.<wbr>access$400(GridDhtAtomicCache.<wbr>java:129)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.distributed.<wbr>dht.atomic.GridDhtAtomicCache$<wbr>5.apply(GridDhtAtomicCache.<wbr>java:265)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.distributed.<wbr>dht.atomic.GridDhtAtomicCache$<wbr>5.apply(GridDhtAtomicCache.<wbr>java:260)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.<wbr>GridCacheIoManager.<wbr>processMessage(<wbr>GridCacheIoManager.java:1060)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.<wbr>GridCacheIoManager.onMessage0(<wbr>GridCacheIoManager.java:579)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.<wbr>GridCacheIoManager.<wbr>handleMessage(<wbr>GridCacheIoManager.java:378)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.<wbr>GridCacheIoManager.<wbr>handleMessage(<wbr>GridCacheIoManager.java:304)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.<wbr>GridCacheIoManager.access$100(<wbr>GridCacheIoManager.java:99)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.<wbr>GridCacheIoManager$1.<wbr>onMessage(GridCacheIoManager.<wbr>java:293)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>managers.communication.<wbr>GridIoManager.invokeListener(<wbr>GridIoManager.java:1555)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>managers.communication.<wbr>GridIoManager.<wbr>processRegularMessage0(<wbr>GridIoManager.java:1183)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>managers.communication.<wbr>GridIoManager.access$4200(<wbr>GridIoManager.java:126)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>managers.communication.<wbr>GridIoManager$9.run(<wbr>GridIoManager.java:1090)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>util.StripedExecutor$Stripe.<wbr>run(StripedExecutor.java:505)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; at java.lang.Thread.run(Thread.<wbr>java:748) [?:1.8.0_151]</div><div>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Suppressed: java.lang.OutOfMemoryError:
Java heap space</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.<wbr>IncompleteCacheObject.&lt;init&gt;(<wbr>IncompleteCacheObject.java:44)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cacheobject.<wbr>IgniteCacheObjectProcessorImpl<wbr>.toCacheObject(<wbr>IgniteCacheObjectProcessorImpl<wbr>.java:191)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.persistence.<wbr>CacheDataRowAdapter.<wbr>readIncompleteValue(<wbr>CacheDataRowAdapter.java:404)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.persistence.<wbr>CacheDataRowAdapter.<wbr>readFragment(<wbr>CacheDataRowAdapter.java:248)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.persistence.<wbr>CacheDataRowAdapter.<wbr>initFromLink(<wbr>CacheDataRowAdapter.java:174)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.persistence.<wbr>CacheDataRowAdapter.<wbr>initFromLink(<wbr>CacheDataRowAdapter.java:102)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.query.h2.database.<wbr>H2RowFactory.getRow(<wbr>H2RowFactory.java:62)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.query.h2.database.<wbr>io.H2ExtrasInnerIO.<wbr>getLookupRow(H2ExtrasInnerIO.<wbr>java:114)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.query.h2.database.<wbr>io.H2ExtrasInnerIO.<wbr>getLookupRow(H2ExtrasInnerIO.<wbr>java:36)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.query.h2.database.<wbr>H2Tree.getRow(H2Tree.java:123)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.query.h2.database.<wbr>H2Tree.getRow(H2Tree.java:40)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.persistence.<wbr>tree.BPlusTree.getRow(<wbr>BPlusTree.java:4372)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.query.h2.database.<wbr>H2Tree.compare(H2Tree.java:<wbr>200)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.query.h2.database.<wbr>H2Tree.compare(H2Tree.java:40)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.persistence.<wbr>tree.BPlusTree.compare(<wbr>BPlusTree.java:4359)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.persistence.<wbr>tree.BPlusTree.<wbr>findInsertionPoint(BPlusTree.<wbr>java:4279)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.persistence.<wbr>tree.BPlusTree.access$1500(<wbr>BPlusTree.java:81)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.persistence.<wbr>tree.BPlusTree$Search.run0(<wbr>BPlusTree.java:261)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.persistence.<wbr>tree.BPlusTree$GetPageHandler.<wbr>run(BPlusTree.java:4697)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.persistence.<wbr>tree.BPlusTree$GetPageHandler.<wbr>run(BPlusTree.java:4682)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.persistence.<wbr>tree.util.PageHandler.<wbr>readPage(PageHandler.java:158)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.persistence.<wbr>DataStructure.read(<wbr>DataStructure.java:319)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.persistence.<wbr>tree.BPlusTree.removeDown(<wbr>BPlusTree.java:1823)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.persistence.<wbr>tree.BPlusTree.removeDown(<wbr>BPlusTree.java:1842)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.persistence.<wbr>tree.BPlusTree.doRemove(<wbr>BPlusTree.java:1752)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.persistence.<wbr>tree.BPlusTree.remove(<wbr>BPlusTree.java:1578)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.query.h2.database.<wbr>H2TreeIndex.remove(<wbr>H2TreeIndex.java:216)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.query.h2.opt.<wbr>GridH2Table.doUpdate(<wbr>GridH2Table.java:504)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.query.h2.opt.<wbr>GridH2Table.update(<wbr>GridH2Table.java:423)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.query.h2.<wbr>IgniteH2Indexing.remove(<wbr>IgniteH2Indexing.java:580)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.query.<wbr>GridQueryProcessor.remove(<wbr>GridQueryProcessor.java:2334)
~[stormjar.jar:?]</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; at org.apache.ignite.internal.<wbr>processors.cache.query.<wbr>GridCacheQueryManager.remove(<wbr>GridCacheQueryManager.java:<wbr>461)</div></div><div><br></div><div><span><br></span></div><div><span><br></span></div>
<div><span><br></span></div>
<div><span><br></span></div>
<div id="m_2476543516489910516ntes-pcmail-signature" style="font-family:'\005fae\008f6f\0096c5\009ed1'">
    

                <div style="font-size:14px;padding:0;margin:0">
                    <div style="font-family:&quot;\005fae\008f6f\0096c5\009ed1&quot;;font-size:13px;color:#000000">
    
<div id="m_2476543516489910516ntes-pcmail-signature-default" style="font-size:14px;color:#000;text-decoration:none">Thanks</div><span
class="HOEnZb"><font color="#888888"><div id="m_2476543516489910516ntes-pcmail-signature-default"
style="font-size:14px;color:#000;text-decoration:none">Shawn</div>
</font></span></div>
                </div>

</div><br>

</div>
</div>
</blockquote></div><br><br clear="all"><div><br></div>--
<br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Best
regards,<br>Andrey V. Mashenkov<br></div></div></div>
</div>
</blockquote><!--😀-->
</div>
</body>
</html>
Mime
View raw message