ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@gridgain.com>
Subject Re: How to overcome short-time network problems?
Date Fri, 08 Jul 2016 07:30:00 GMT
Kamal,

It’s preferable then to tune GC settings to avoid such long stop the world pauses. After
this is done there is no need to take care of the segmentation policy at all.

Read this [1] article which provides general recommendations on how to tune the GC and JVM.
Also keep in mind if your caches are big in size occupying around 20 GB of Java heap in total
then it definitely makes sense to switch to OFFHEAP_TIRED mode.

[1] https://apacheignite.readme.io/docs/jvm-and-system-tuning

—
Denis

> On Jul 8, 2016, at 9:36 AM, Kamal C <kamaltarget@gmail.com> wrote:
> 
> Thanks for your response Denis. 
> 
> Sorry for the late reply. I'm starting the Ignite node in embedded mode.
> RESTART_JVM segmentation policy is applicable for standalone Ignite 
> node. Is there any other way? (It's happening once in a while I'm not able
> to find out it's root cause)
> 
> In our application, during peak times Full GC can take upto 40-70 seconds.
> How to configure session / connection timeout ?
> 
> --
> Kamal
> 
> On Thu, Jun 30, 2016 at 7:04 PM, Denis Magda <dmagda@gridgain.com <mailto:dmagda@gridgain.com>>
wrote:
> Hi Kamal,
> 
> There are several options how this kind of exception may be processed by the kernel.
See org.apache.ignite.plugin.segmentation.SegmentationPolicy for more details. 
> 
> In your scenario you can set IgniteConfiguration.setSegmentationPolicy(SegmentationPolicy.RESTART_JVM).
> 
> In any case are you sure that the reason of the issue is related to network problems?
I would suggest to check the logs of the node that was segmented for more details. Also usually
the node become segmented due to long garbage collection. Read this page [1] and make sure
that you’re not facing with the stop-the-world pauses.
> 
> [1] https://apacheignite.readme.io/docs/jvm-and-system-tuning <https://apacheignite.readme.io/docs/jvm-and-system-tuning>
> 
> —
> Denis
> 
>> On Jun 30, 2016, at 2:18 PM, Kamal C <kamaltarget@gmail.com <mailto:kamaltarget@gmail.com>>
wrote:
>> 
>> Hi,
>> 
>>     In my setup, sometimes due to network problems local node gets segmented and
stopped. 
>> How to tell Ignite to retry continuously to connect with peers instead of stopping
the node ?
>> 
>> Logs:
>> 
>> WARN [2016-06-30T16:26:15,370] TcpDiscoverySpi: warning(): Node is out of topology
(probably, due to short-time network problems).
>> WARN [2016-06-30T16:26:15,370] GridDiscoveryManager: warning(): Local node SEGMENTED:
TcpDiscoveryNode [id=7d24ed24-a742-4d22-b52a-4ce33317e209, addrs=[192.168.11.127], sockAddrs=[tcltestvm3.nmsworks.co.in/192.168.11.127:47500
<http://tcltestvm3.nmsworks.co.in/192.168.11.127:47500>, /192.168.11.127:47500 <http://192.168.11.127:47500/>],
discPort=47500, order=2, intOrder=2, lastExchangeTime=1467284175362, loc=true, ver=1.6.0#19700101-sha1:00000000,
isClient=false]
>> WARN [2016-06-30T16:26:15,417] GridDiscoveryManager: warning(): Stopping local node
according to configured segmentation policy.
>> WARN [2016-06-30T16:26:15,418] GridDiscoveryManager: warning(): Node FAILED: TcpDiscoveryNode
[id=1a2bfcd9-d72f-4a1e-a69f-c6d0b8f8814f, addrs=[192.168.9.211], sockAddrs=[tcltest1.nmsworks.co.in/192.168.9.211:47500
<http://tcltest1.nmsworks.co.in/192.168.9.211:47500>, /192.168.9.211:47500 <http://192.168.9.211:47500/>],
discPort=47500, order=1, intOrder=1, lastExchangeTime=1467278442434, loc=false, ver=1.6.0#19700101-sha1:00000000,
isClient=false]
>> 
>> 
>> Regards,
>> Kamal C
> 
> 


Mime
View raw message