ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "shawn.du" <shawn...@neulion.com.cn>
Subject Re: Ignite server send big data to client and make client crash
Date Sat, 12 Aug 2017 08:23:29 GMT
<html>
<head>
    <meta http-equiv='Content-Type' content='text/html; charset=UTF-8'>
</head>
<body>
<style>
    font{
        line-height: 1.5;
    }
</style>
<div style = 'font-family:"微软雅黑"; font-size: 14px; color:#000000; line-height:1.5;'>
    <div>
<div><span>BTW,</span></div><div><span><br></span></div><div>We
also use ignite file system and message pub/sub technology. will they do some sync which may
cause high cpu/traffic?</div>
<div><br></div>
<div id="ntes-pcmail-signature" style="font-family:'微软雅黑'">
    <style type="text/css">
        a#ntes-pcmail-signature-default:hover {
            text-decoration: underline;
            color: #3593db;
            cursor: pointer;
        }
    </style>

                <div style="font-size:14px; padding: 0;  margin:0;">
                    <div style="font-family:&quot;微软雅黑&quot;; font-size:
13px; color:#000000">
    <style>
        font{
            line-height: 1.5;
        }
    </style>
<div id="ntes-pcmail-signature-default" style="font-size:14px; color:#000; text-decoration:
none;">Thanks</div><div id="ntes-pcmail-signature-default" style="font-size:14px;
color:#000; text-decoration: none;">Shawn</div>
</div>
                </div>

</div><br>
</div><div class="J-reply" style="background-color:#f2f2f2;color:black;padding-top:6px;padding-bottom:6px;border-radius:3px;-moz-border-radius:3px;-webkit-border-radius:3px;margin-top:45px;margin-bottom:20px;font-family:'微软雅黑';">
    <div style="font-size:14px;line-height:1.5;word-break:break-all;margin-left:10px;margin-right:10px">On
<span class="mail-date">08/12/2017 08:04</span>,<a class="mail-to" style="text-decoration:none;color:#2a97ff;"
href="mailto:shawn.du@neulion.com.cn">shawn.du&lt;shawn.du@neulion.com.cn&gt;</a>
wrote: </div>
</div>
<blockquote id="ntes-pcmail-quote" style="margin: 0; padding: 0; font-size: 13px; font-family:
'微软雅黑';">

<style>
    font{
        line-height: 1.5;
    }
</style>
<div style="font-family:&quot;微软雅黑&quot;; font-size: 14px; color:#000000;
line-height:1.5;">
    <div>
<div><span>Hi,</span></div><div><span><br></span></div><div><span>No,
still in&nbsp;investigation. &nbsp;</span></div><div><br></div><div>This
is the CPU utilization of ignite server. You can see that there are 3 points which are bigger
that others.</div><div><br></div><div>Ignite client &nbsp;disconnected
at these time. In fact we use ignite in storm, this will cause storm worker stopped and restart.</div><div><br></div><div><img
src="cid:FB7042E8-C6A9-491E-945B-9D9ABCAFA383@mailmaster"></div>
<div><span><br></span></div><div><img src="cid:4A8D8B8B-0953-462C-9C86-389DAB27B4F1@mailmaster"></div>
<div><span><br></span></div><div>We suspect below code,
but not sure.</div><div><br></div><div>The Interceptor we persist
cache entry into file in json format. &nbsp;there is a rotate policy. currently We use
time-based policy,</div><div>so file will be moved to another place hourly. This
is the reason in the network-out-bytes chart, some big points appears hourly.</div><div>But
not all of them cause by the interceptor. &nbsp;The 2nd and the 3rd red point are exceptions.
I don't know why they occurred now.</div><div><br></div><div>My
question is the CPU is not so high, why client is disconnected from server.&nbsp;</div><div><br></div><div>Thanks
in advance.</div><div>=====================code=============================</div><div><div>public
class PersistJsonInterceptor&lt;K, V&gt; extends CacheInterceptorAdapter&lt;K,
V&gt;</div><div>{</div><div>&nbsp; &nbsp; @IgniteInstanceResource</div><div>&nbsp;
&nbsp; private Ignite ignite;</div><div>&nbsp; &nbsp; @LoggerResource</div><div>&nbsp;
&nbsp; private IgniteLogger logger</div><div>&nbsp; &nbsp; private
transient BlockingQueue&lt;V&gt; queue = new LinkedBlockingQueue&lt;&gt;();</div><div>&nbsp;
&nbsp; public void onAfterRemove(Cache.Entry&lt;K, V&gt; entry)</div><div>&nbsp;
&nbsp; {</div><div><span class="Apple-tab-span" style="white-space:pre">
</span>if (worker == null)</div><div>&nbsp; &nbsp; &nbsp; &nbsp;
{</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
...</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
worker = new WriteWorker(localTmpDir, destDir, partitionId, policy);</div><div>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; new Thread(worker).start();</div><div>&nbsp;
&nbsp; &nbsp; &nbsp; }</div><div>&nbsp; &nbsp; &nbsp;
&nbsp; ...</div><div>&nbsp; &nbsp; &nbsp; &nbsp; if (value
== null)</div><div>&nbsp; &nbsp; &nbsp; &nbsp; {</div><div>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; return;</div><div>&nbsp;
&nbsp; &nbsp; &nbsp; }</div><div>&nbsp; &nbsp; &nbsp;
&nbsp; queue.add(value);</div><div>&nbsp; &nbsp; }</div><div><span
class="Apple-tab-span" style="white-space:pre">	</span></div><div>&nbsp;
&nbsp;class WriteWorker implements Runnable</div><div>&nbsp; {</div><div><span
class="Apple-tab-span" style="white-space:pre">	</span>@Override</div><div>&nbsp;
&nbsp; &nbsp; &nbsp; public void run()</div><div>&nbsp; &nbsp;
&nbsp; &nbsp; {</div><div>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; while (true)</div><div>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; {</div><div>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; try</div><div>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; {</div><div>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
V value = queue.take();</div><div>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; init(value.getClass());</div><div>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
objectMapper.writeValue(fileWriter, value);</div><div>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; fileWriter.append("\n");</div><div>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
if (policy.rotateReady(this.file))</div><div>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; {</div><div><span
class="Apple-tab-span" style="white-space:pre">		</span>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; ...</div><div><span class="Apple-tab-span" style="white-space:pre">
		</span>&nbsp;&nbsp;this.fileWriter.close();</div><div><span
class="Apple-tab-span" style="white-space:pre">			</span>&nbsp;&nbsp;policy.rotate(this.file,
new File(this.destDir), partitionId);</div><div>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
...</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; }</div><div>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }</div><div>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; catch (Exception
e)</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; {</div><div>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; logger.error("fail to persist
data into json file ...", e);</div><div>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; }</div><div>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; }</div><div>&nbsp; &nbsp;
&nbsp; &nbsp; }</div><div>&nbsp; &nbsp; }</div><div>}</div></div><div><br></div>
<div id="ntes-pcmail-signature" style="font-family:'微软雅黑'">
    <style type="text/css">
        a#ntes-pcmail-signature-default:hover {
            text-decoration: underline;
            color: #3593db;
            cursor: pointer;
        }
    </style>

                <div style="font-size:14px; padding: 0;  margin:0;">
                    <div style="font-family:&quot;微软雅黑&quot;; font-size:
13px; color:#000000">
    <style>
        font{
            line-height: 1.5;
        }
    </style>
<div id="ntes-pcmail-signature-default" style="font-size:14px; color:#000; text-decoration:
none;">Thanks</div><div id="ntes-pcmail-signature-default" style="font-size:14px;
color:#000; text-decoration: none;">Shawn</div>
</div>
                </div>

</div><br>
</div><div class="J-reply" style="background-color:#f2f2f2;color:black;padding-top:6px;padding-bottom:6px;border-radius:3px;-moz-border-radius:3px;-webkit-border-radius:3px;margin-top:45px;margin-bottom:20px;font-family:'微软雅黑';">
    <div style="font-size:14px;line-height:1.5;word-break:break-all;margin-left:10px;margin-right:10px">On
<span class="mail-date">08/11/2017 15:59</span>,<a class="mail-to" style="text-decoration:none;color:#2a97ff;"
href="mailto:mcherkasov@gridgain.com">Mikhail Cherkasov&lt;mcherkasov@gridgain.com&gt;</a>
wrote: </div>
</div>
<blockquote id="ntes-pcmail-quote" style="margin: 0; padding: 0; font-size: 13px; font-family:
'微软雅黑';">
<div dir="ltr">Hi Shawn,<div><br></div><div>Did you figure out
what was wrong?</div><div><br></div><div>Thanks,</div><div>Mikhail.</div><div><br></div></div><div
class="gmail_extra"><br><div class="gmail_quote">On Fri, Aug 11, 2017 at 9:31
AM, shawn.du <span dir="ltr">&lt;<a href="mailto:shawn.du@neulion.com.cn" target="_blank">shawn.du@neulion.com.cn</a>&gt;</span>
wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">

    

<div>

<div style="font-family:&quot;\005fae\008f6f\0096c5\009ed1&quot;;font-size:14px;color:#000000;line-height:1.5">
    <div>
<div><span><br></span></div>
<div>sorry, Ignore this.</div>
<div><span><br></span></div>
<div id="m_1333560839707655551ntes-pcmail-signature" style="font-family:'\005fae\008f6f\0096c5\009ed1'">
    

                <div style="font-size:14px;padding:0;margin:0">
                    <div style="font-family:&quot;\005fae\008f6f\0096c5\009ed1&quot;;font-size:13px;color:#000000">
    
<div id="m_1333560839707655551ntes-pcmail-signature-default" style="font-size:14px;color:#000;text-decoration:none">Thanks</div><span
class="HOEnZb"><font color="#888888"><div id="m_1333560839707655551ntes-pcmail-signature-default"
style="font-size:14px;color:#000;text-decoration:none">Shawn</div>
</font></span></div>
                </div>

</div><br>
</div><div><div class="h5"><div class="m_1333560839707655551J-reply"
style="background-color:#f2f2f2;color:black;padding-top:6px;padding-bottom:6px;border-radius:3px;margin-top:45px;margin-bottom:20px;font-family:'\005fae\008f6f\0096c5\009ed1'">
    <div style="font-size:14px;line-height:1.5;word-break:break-all;margin-left:10px;margin-right:10px">On
<span class="m_1333560839707655551mail-date">08/11/2017 11:03</span>,<a class="m_1333560839707655551mail-to"
style="text-decoration:none;color:#2a97ff" href="mailto:shawn.du@neulion.com.cn" target="_blank">shawn.du&lt;shawn.du@<wbr>neulion.com.cn&gt;</a>
wrote: </div>
</div>
<blockquote id="m_1333560839707655551ntes-pcmail-quote" style="margin:0;padding:0;font-size:13px;font-family:'\005fae\008f6f\0096c5\009ed1'">


<div style="font-family:&quot;\005fae\008f6f\0096c5\009ed1&quot;;font-size:14px;color:#000000;line-height:1.5">
    <div><div>Hi community,</div><div><br></div><div>I
have a ignite server(1.9.0) and a ignite client.</div><div><br></div><div>Recently
my client crashed 4 times.&nbsp; I found it may be caused by server send big data to client.&nbsp;</div><div><br></div><div>see
blow picture. &nbsp;this is AWS Cloud Watch network-out-bytes picture of ignite server.&nbsp;</div><div><br></div><div>It
seems that ignite server do something hourly.&nbsp; My client crashed at the red points.</div><div>one
time is 19:33, the other is 23:56.</div><div><br></div></div><div>is
there any settings to avoid this?</div><div><br></div><div>Thanks
in advance.</div><div><br></div><div><img src="cid:BEC4F645-C470-4CE0-BBA5-DF205D72D00C@mailmaster"></div><div><br></div>
<div id="m_1333560839707655551ntes-pcmail-signature" style="font-family:'\005fae\008f6f\0096c5\009ed1'">
    

                <div style="font-size:14px;padding:0;margin:0">
                    <div style="font-family:&quot;\005fae\008f6f\0096c5\009ed1&quot;;font-size:13px;color:#000000">
    
<div id="m_1333560839707655551ntes-pcmail-signature-default" style="font-size:14px;color:#000;text-decoration:none">Thanks</div><div
id="m_1333560839707655551ntes-pcmail-signature-default" style="font-size:14px;color:#000;text-decoration:none">Shawn</div>
</div>
                </div>

</div><br>

</div></blockquote>
</div></div></div>
</div>
</blockquote></div><br><br clear="all"><div><br></div>--
<br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Thanks,<div>Mikhail.</div></div></div>
</div>
</blockquote><!--😀-->
</div></blockquote><!--😀-->
</div>
</body>
</html>
Mime
View raw message