ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aurora <2565003...@qq.com>
Subject Re: cluster hanged when client node put data into the caches
Date Sun, 10 Dec 2017 13:49:16 GMT
@Nikolai
Your suggestion is right.
"to-space exhausted" occured  in GC Logs on 3 out of 6 server nodes, as:
Node 1:
2017-12-05T18:36:48.716+0800: 6870.705: [GC pause (G1 Evacuation Pause)
(young), 0.4171797 secs]
   [Parallel Time: 412.1 ms, GC Workers: 13]
      [GC Worker Start (ms): Min: 6870705.5, Avg: 6870705.7, Max: 6870705.8,
Diff: 0.3]
      [Ext Root Scanning (ms): Min: 0.6, Avg: 0.7, Max: 0.9, Diff: 0.4, Sum:
9.6]
      [Update RS (ms): Min: 56.8, Avg: 58.6, Max: 60.2, Diff: 3.4, Sum:
761.8]
         [Processed Buffers: Min: 69, Avg: 76.7, Max: 91, Diff: 22, Sum:
997]
      [Scan RS (ms): Min: 1.4, Avg: 2.8, Max: 4.7, Diff: 3.3, Sum: 36.7]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1,
Sum: 0.2]
      [Object Copy (ms): Min: 349.1, Avg: 349.2, Max: 349.3, Diff: 0.2, Sum:
4540.2]
      [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.5]
         [Termination Attempts: Min: 1, Avg: 40.3, Max: 75, Diff: 74, Sum:
524]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.2, Sum:
1.7]
      [GC Worker Total (ms): Min: 411.4, Avg: 411.6, Max: 411.8, Diff: 0.4,
Sum: 5350.7]
      [GC Worker End (ms): Min: 6871117.2, Avg: 6871117.3, Max: 6871117.4,
Diff: 0.2]
   [Code Root Fixup: 0.1 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.9 ms]
   [Other: 4.1 ms]
      [Choose CSet: 0.0 ms]
      [Ref Proc: 0.4 ms]
      [Ref Enq: 0.0 ms]
      [Redirty Cards: 0.9 ms]
      [Humongous Register: 0.1 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 1.9 ms]
   [Eden: 8280.0M(8280.0M)->0.0B(8032.0M) Survivors: 968.0M->968.0M Heap:
13.4G(16.0G)->5680.8M(16.0G)]
 [Times: user=5.37 sys=0.00, real=0.42 secs] 
2017-12-05T18:40:45.218+0800: 7107.207: [GC pause (G1 Evacuation Pause)
(young) (to-space exhausted), 19.5379052 secs]
   [Parallel Time: 16666.0 ms, GC Workers: 13]
      [GC Worker Start (ms): Min: 7107207.2, Avg: 7107207.4, Max: 7107207.5,
Diff: 0.3]
      [Ext Root Scanning (ms): Min: 0.6, Avg: 0.7, Max: 0.9, Diff: 0.3, Sum:
9.1]
      [Update RS (ms): Min: 143.9, Avg: 144.0, Max: 144.1, Diff: 0.2, Sum:
1871.6]
         [Processed Buffers: Min: 78, Avg: 87.2, Max: 108, Diff: 30, Sum:
1134]
      [Scan RS (ms): Min: 6.1, Avg: 6.2, Max: 6.3, Diff: 0.2, Sum: 80.3]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0,
Sum: 0.2]
      [Object Copy (ms): Min: 16474.9, Avg: 16481.7, Max: 16514.7, Diff:
39.8, Sum: 214262.3]
      [Termination (ms): Min: 0.0, Avg: 32.9, Max: 39.6, Diff: 39.6, Sum:
427.8]
         [Termination Attempts: Min: 1, Avg: 1.5, Max: 4, Diff: 3, Sum: 19]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum:
0.9]
      [GC Worker Total (ms): Min: 16665.4, Avg: 16665.6, Max: 16665.7, Diff:
0.3, Sum: 216652.3]
      [GC Worker End (ms): Min: 7123872.9, Avg: 7123872.9, Max: 7123873.0,
Diff: 0.1]
   [Code Root Fixup: 0.1 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.8 ms]
   [Other: 2871.0 ms]
      [Evacuation Failure: 2861.6 ms]
      [Choose CSet: 0.0 ms]
      [Ref Proc: 0.4 ms]
      [Ref Enq: 0.0 ms]
      [Redirty Cards: 6.9 ms]
      [Humongous Register: 0.1 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 1.3 ms]
   [Eden: 8032.0M(8032.0M)->0.0B(328.0M) Survivors: 968.0M->1504.0M Heap:
13.5G(16.0G)->12.5G(16.0G)]
 [Times: user=43.38 sys=7.36, real=19.53 secs] 
2017-12-05T18:41:05.194+0800: 7127.183: [GC pause (G1 Evacuation Pause)
(young) (initial-mark), 0.9987695 secs]
   [Parallel Time: 990.4 ms, GC Workers: 13]
      [GC Worker Start (ms): Min: 7127183.8, Avg: 7127183.9, Max: 7127184.0,
Diff: 0.2]
      [Ext Root Scanning (ms): Min: 0.9, Avg: 1.1, Max: 1.3, Diff: 0.4, Sum:
14.3]
      [Update RS (ms): Min: 563.4, Avg: 563.6, Max: 564.4, Diff: 1.0, Sum:
7327.2]
         [Processed Buffers: Min: 1025, Avg: 1339.8, Max: 1575, Diff: 550,
Sum: 17418]
      [Scan RS (ms): Min: 15.0, Avg: 15.7, Max: 16.0, Diff: 1.0, Sum: 204.6]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0,
Sum: 0.1]
      [Object Copy (ms): Min: 409.2, Avg: 409.4, Max: 409.4, Diff: 0.2, Sum:
5321.6]
      [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.6]
         [Termination Attempts: Min: 1, Avg: 33.3, Max: 48, Diff: 47, Sum:
433]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum:
1.2]
      [GC Worker Total (ms): Min: 989.8, Avg: 990.0, Max: 990.2, Diff: 0.4,
Sum: 12869.6]
      [GC Worker End (ms): Min: 7128173.8, Avg: 7128173.9, Max: 7128174.0,
Diff: 0.2]
   [Code Root Fixup: 0.1 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.3 ms]
   [Other: 7.9 ms]
      [Choose CSet: 0.0 ms]
      [Ref Proc: 0.4 ms]
      [Ref Enq: 0.0 ms]
      [Redirty Cards: 4.8 ms]
      [Humongous Register: 0.0 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 0.8 ms]
   [Eden: 328.0M(328.0M)->0.0B(336.0M) Survivors: 1504.0M->280.0M Heap:
12.9G(16.0G)->12.8G(16.0G)]
 [Times: user=12.93 sys=0.00, real=1.00 secs] 

Node 2:
2017-12-05T18:37:14.177+0800: 9497.100: [GC pause (G1 Evacuation Pause)
(young), 0.2761496 secs]
   [Parallel Time: 271.9 ms, GC Workers: 13]
      [GC Worker Start (ms): Min: 9497100.0, Avg: 9497100.2, Max: 9497100.4,
Diff: 0.3]
      [Ext Root Scanning (ms): Min: 0.6, Avg: 0.8, Max: 0.9, Diff: 0.4, Sum:
10.0]
      [Update RS (ms): Min: 19.0, Avg: 20.3, Max: 22.0, Diff: 3.0, Sum:
264.3]
         [Processed Buffers: Min: 6, Avg: 13.7, Max: 42, Diff: 36, Sum: 178]
      [Scan RS (ms): Min: 0.1, Avg: 0.3, Max: 0.7, Diff: 0.7, Sum: 3.5]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1,
Sum: 0.2]
      [Object Copy (ms): Min: 247.7, Avg: 249.3, Max: 250.5, Diff: 2.7, Sum:
3241.1]
      [Termination (ms): Min: 0.0, Avg: 0.5, Max: 0.7, Diff: 0.7, Sum: 6.9]
         [Termination Attempts: Min: 1, Avg: 374.3, Max: 476, Diff: 475,
Sum: 4866]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.3, Sum:
2.1]
      [GC Worker Total (ms): Min: 271.1, Avg: 271.4, Max: 271.7, Diff: 0.6,
Sum: 3528.1]
      [GC Worker End (ms): Min: 9497371.5, Avg: 9497371.6, Max: 9497371.7,
Diff: 0.3]
   [Code Root Fixup: 0.1 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.9 ms]
   [Other: 3.2 ms]
      [Choose CSet: 0.0 ms]
      [Ref Proc: 0.3 ms]
      [Ref Enq: 0.0 ms]
      [Redirty Cards: 0.4 ms]
      [Humongous Register: 0.1 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 2.0 ms]
   [Eden: 9392.0M(9392.0M)->0.0B(9080.0M) Survivors: 432.0M->744.0M Heap:
13.5G(16.0G)->4752.0M(16.0G)]
 [Times: user=3.54 sys=0.01, real=0.28 secs] 
2017-12-05T18:43:29.983+0800: 9872.905: [GC pause (G1 Evacuation Pause)
(young) (to-space exhausted), 45.1101748 secs]
   [Parallel Time: 39495.6 ms, GC Workers: 13]
      [GC Worker Start (ms): Min: 9872905.5, Avg: 9872905.7, Max: 9872905.8,
Diff: 0.3]
      [Ext Root Scanning (ms): Min: 0.6, Avg: 0.8, Max: 0.9, Diff: 0.3, Sum:
9.8]
      [Update RS (ms): Min: 1180.8, Avg: 1183.2, Max: 1184.5, Diff: 3.7,
Sum: 15382.2]
         [Processed Buffers: Min: 63, Avg: 73.8, Max: 95, Diff: 32, Sum:
960]
      [Scan RS (ms): Min: 0.1, Avg: 0.2, Max: 0.4, Diff: 0.4, Sum: 2.8]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0,
Sum: 0.2]
      [Object Copy (ms): Min: 38287.7, Avg: 38290.6, Max: 38310.0, Diff:
22.3, Sum: 497777.7]
      [Termination (ms): Min: 0.0, Avg: 20.3, Max: 22.0, Diff: 22.0, Sum:
263.9]
         [Termination Attempts: Min: 1, Avg: 20.8, Max: 32, Diff: 31, Sum:
271]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum:
1.3]
      [GC Worker Total (ms): Min: 39495.0, Avg: 39495.2, Max: 39495.5, Diff:
0.4, Sum: 513437.7]
      [GC Worker End (ms): Min: 9912400.8, Avg: 9912400.9, Max: 9912401.0,
Diff: 0.2]
   [Code Root Fixup: 0.1 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.8 ms]
   [Other: 5613.6 ms]
      [Evacuation Failure: 5603.4 ms]
      [Choose CSet: 0.0 ms]
      [Ref Proc: 0.4 ms]
      [Ref Enq: 0.0 ms]
      [Redirty Cards: 8.1 ms]
      [Humongous Register: 0.1 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 1.0 ms]
   [Eden: 9080.0M(9080.0M)->0.0B(8192.0K) Survivors: 744.0M->1640.0M Heap:
13.6G(16.0G)->13.2G(16.0G)]
 [Times: user=75.59 sys=19.17, real=45.10 secs] 
2017-12-05T18:44:15.116+0800: 9918.039: [GC pause (G1 Evacuation Pause)
(young) (initial-mark), 1.2374672 secs]
   [Parallel Time: 1229.6 ms, GC Workers: 13]
      [GC Worker Start (ms): Min: 9918039.4, Avg: 9918039.5, Max: 9918039.6,
Diff: 0.2]
      [Ext Root Scanning (ms): Min: 0.8, Avg: 0.9, Max: 1.2, Diff: 0.4, Sum:
12.3]
      [Update RS (ms): Min: 855.4, Avg: 855.5, Max: 855.7, Diff: 0.4, Sum:
11121.4]
         [Processed Buffers: Min: 1844, Avg: 2442.1, Max: 2840, Diff: 996,
Sum: 31747]
      [Scan RS (ms): Min: 13.3, Avg: 13.4, Max: 13.5, Diff: 0.3, Sum: 174.2]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0,
Sum: 0.1]
      [Object Copy (ms): Min: 359.2, Avg: 359.2, Max: 359.3, Diff: 0.1, Sum:
4669.9]
      [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.2]
         [Termination Attempts: Min: 1, Avg: 19.1, Max: 33, Diff: 32, Sum:
248]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum:
1.4]
      [GC Worker Total (ms): Min: 1229.0, Avg: 1229.2, Max: 1229.4, Diff:
0.4, Sum: 15979.5]
      [GC Worker End (ms): Min: 9919268.6, Avg: 9919268.7, Max: 9919268.8,
Diff: 0.2]
   [Code Root Fixup: 0.1 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.3 ms]
   [Other: 7.4 ms]
      [Choose CSet: 0.0 ms]
      [Ref Proc: 0.4 ms]
      [Ref Enq: 0.0 ms]
      [Redirty Cards: 4.7 ms]
      [Humongous Register: 0.0 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 0.8 ms]
   [Eden: 8192.0K(8192.0K)->0.0B(456.0M) Survivors: 1640.0M->56.0M Heap:
13.2G(16.0G)->13.3G(16.0G)]
 [Times: user=16.03 sys=0.00, real=1.24 secs] 
2017-12-05T18:44:16.354+0800: 9919.276: [GC
concurrent-root-region-scan-start]
2017-12-05T18:44:16.368+0800: 9919.291: [GC concurrent-root-region-scan-end,
0.0141460 secs]
2017-12-05T18:44:16.368+0800: 9919.291: [GC concurrent-mark-start]
2017-12-05T18:44:19.218+0800: 9922.141: [GC pause (G1 Evacuation Pause)
(young), 0.1909109 secs]
   [Parallel Time: 187.1 ms, GC Workers: 13]
      [GC Worker Start (ms): Min: 9922141.8, Avg: 9922141.9, Max: 9922142.1,
Diff: 0.3]
      [Ext Root Scanning (ms): Min: 0.6, Avg: 0.7, Max: 0.9, Diff: 0.3, Sum:
9.6]
      [Update RS (ms): Min: 54.0, Avg: 54.2, Max: 54.5, Diff: 0.5, Sum:
704.3]
         [Processed Buffers: Min: 58, Avg: 64.4, Max: 80, Diff: 22, Sum:
837]
      [Scan RS (ms): Min: 4.9, Avg: 5.2, Max: 5.4, Diff: 0.5, Sum: 68.1]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0,
Sum: 0.0]
      [Object Copy (ms): Min: 126.3, Avg: 126.3, Max: 126.4, Diff: 0.2, Sum:
1641.9]
      [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 1.6]
         [Termination Attempts: Min: 1, Avg: 77.5, Max: 97, Diff: 96, Sum:
1007]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum:
1.0]
      [GC Worker Total (ms): Min: 186.5, Avg: 186.7, Max: 186.9, Diff: 0.4,
Sum: 2426.6]
      [GC Worker End (ms): Min: 9922328.6, Avg: 9922328.6, Max: 9922328.7,
Diff: 0.1]
   [Code Root Fixup: 0.1 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.3 ms]
   [Other: 3.4 ms]
      [Choose CSet: 0.0 ms]
      [Ref Proc: 0.5 ms]
      [Ref Enq: 0.0 ms]
      [Redirty Cards: 1.2 ms]
      [Humongous Register: 0.1 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 0.4 ms]
   [Eden: 456.0M(456.0M)->0.0B(424.0M) Survivors: 56.0M->88.0M Heap:
13.8G(16.0G)->13.7G(16.0G)]
 [Times: user=2.43 sys=0.01, real=0.19 secs] 

Node 3:
2017-12-05T18:41:07.782+0800: 7118.035: [GC pause (G1 Evacuation Pause)
(young), 0.5114032 secs]
   [Parallel Time: 506.9 ms, GC Workers: 13]
      [GC Worker Start (ms): Min: 7118034.9, Avg: 7118035.0, Max: 7118035.2,
Diff: 0.3]
      [Ext Root Scanning (ms): Min: 0.6, Avg: 0.7, Max: 0.9, Diff: 0.3, Sum:
9.3]
      [Update RS (ms): Min: 50.6, Avg: 51.5, Max: 52.5, Diff: 1.9, Sum:
669.4]
         [Processed Buffers: Min: 20, Avg: 31.3, Max: 56, Diff: 36, Sum:
407]
      [Scan RS (ms): Min: 0.1, Avg: 0.2, Max: 0.6, Diff: 0.6, Sum: 3.0]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0,
Sum: 0.2]
      [Object Copy (ms): Min: 453.0, Avg: 453.8, Max: 454.3, Diff: 1.3, Sum:
5899.5]
      [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.5]
         [Termination Attempts: Min: 1, Avg: 20.4, Max: 27, Diff: 26, Sum:
265]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.2, Sum:
1.8]
      [GC Worker Total (ms): Min: 506.2, Avg: 506.4, Max: 506.7, Diff: 0.5,
Sum: 6583.7]
      [GC Worker End (ms): Min: 7118541.4, Avg: 7118541.5, Max: 7118541.6,
Diff: 0.2]
   [Code Root Fixup: 0.1 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.9 ms]
   [Other: 3.5 ms]
      [Choose CSet: 0.0 ms]
      [Ref Proc: 0.5 ms]
      [Ref Enq: 0.0 ms]
      [Redirty Cards: 0.5 ms]
      [Humongous Register: 0.1 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 1.8 ms]
   [Eden: 9208.0M(9208.0M)->0.0B(7160.0M) Survivors: 384.0M->1600.0M Heap:
13.5G(16.0G)->5928.0M(16.0G)]
 [Times: user=6.59 sys=0.00, real=0.52 secs] 
2017-12-05T18:59:32.939+0800: 8223.192: [GC pause (G1 Evacuation Pause)
(young) (to-space exhausted), 55.7872972 secs]
   [Parallel Time: 50147.5 ms, GC Workers: 13]
      [GC Worker Start (ms): Min: 8223192.1, Avg: 8223192.2, Max: 8223192.4,
Diff: 0.3]
      [Ext Root Scanning (ms): Min: 0.6, Avg: 0.7, Max: 0.9, Diff: 0.3, Sum:
9.2]
      [Update RS (ms): Min: 514.3, Avg: 516.5, Max: 523.9, Diff: 9.6, Sum:
6714.9]
         [Processed Buffers: Min: 105, Avg: 122.5, Max: 165, Diff: 60, Sum:
1593]
      [Scan RS (ms): Min: 1916.0, Avg: 1923.1, Max: 1925.4, Diff: 9.5, Sum:
25000.0]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2,
Sum: 1.0]
      [Object Copy (ms): Min: 47705.9, Avg: 47706.4, Max: 47707.0, Diff:
1.0, Sum: 620183.4]
      [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.2, Sum: 1.8]
         [Termination Attempts: Min: 1, Avg: 28.2, Max: 47, Diff: 46, Sum:
367]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum:
1.0]
      [GC Worker Total (ms): Min: 50146.9, Avg: 50147.0, Max: 50147.2, Diff:
0.3, Sum: 651911.3]
      [GC Worker End (ms): Min: 8273339.2, Avg: 8273339.3, Max: 8273339.3,
Diff: 0.2]
   [Code Root Fixup: 0.1 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.6 ms]
   [Other: 5639.1 ms]
      [Evacuation Failure: 5622.5 ms]
      [Choose CSet: 0.0 ms]
      [Ref Proc: 0.3 ms]
      [Ref Enq: 0.0 ms]
      [Redirty Cards: 12.1 ms]
      [Humongous Register: 0.1 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 3.7 ms]
   [Eden: 7160.0M(7160.0M)->0.0B(8192.0K) Survivors: 1600.0M->1232.0M Heap:
13.1G(16.0G)->13.8G(16.0G)]
 [Times: user=96.95 sys=24.80, real=55.78 secs] 
2017-12-05T19:00:28.876+0800: 8279.128: [GC pause (G1 Evacuation Pause)
(young) (initial-mark), 2.3525928 secs]
   [Parallel Time: 2345.5 ms, GC Workers: 13]
      [GC Worker Start (ms): Min: 8279129.3, Avg: 8279129.4, Max: 8279129.5,
Diff: 0.2]
      [Ext Root Scanning (ms): Min: 0.7, Avg: 0.8, Max: 1.0, Diff: 0.3, Sum:
10.9]
      [Update RS (ms): Min: 1933.1, Avg: 1933.3, Max: 1933.6, Diff: 0.5,
Sum: 25132.7]
         [Processed Buffers: Min: 2505, Avg: 3215.6, Max: 3919, Diff: 1414,
Sum: 41803]
      [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.5]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1,
Sum: 0.1]
      [Object Copy (ms): Min: 410.4, Avg: 410.7, Max: 410.9, Diff: 0.5, Sum:
5338.6]
      [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.7]
         [Termination Attempts: Min: 1, Avg: 84.6, Max: 120, Diff: 119, Sum:
1100]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum:
1.2]
      [GC Worker Total (ms): Min: 2344.9, Avg: 2345.1, Max: 2345.2, Diff:
0.3, Sum: 30485.8]
      [GC Worker End (ms): Min: 8281474.4, Avg: 8281474.5, Max: 8281474.6,
Diff: 0.2]
   [Code Root Fixup: 0.1 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.4 ms]
   [Other: 6.6 ms]
      [Choose CSet: 0.0 ms]
      [Ref Proc: 0.7 ms]
      [Ref Enq: 0.0 ms]
      [Redirty Cards: 3.5 ms]
      [Humongous Register: 0.0 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 0.4 ms]
   [Eden: 8192.0K(8192.0K)->0.0B(472.0M) Survivors: 1232.0M->40.0M Heap:
13.8G(16.0G)->13.9G(16.0G)]
 [Times: user=30.54 sys=0.00, real=2.35 secs] 
2017-12-05T19:00:31.229+0800: 8281.481: [GC
concurrent-root-region-scan-start]
2017-12-05T19:00:31.230+0800: 8281.482: [GC concurrent-root-region-scan-end,
0.0008827 secs]
2017-12-05T19:00:31.230+0800: 8281.482: [GC concurrent-mark-start]
2017-12-05T19:00:31.332+0800: 8281.585: [GC concurrent-mark-end, 0.1025943
secs]
2017-12-05T19:00:31.334+0800: 8281.586: [GC remark
2017-12-05T19:00:31.334+0800: 8281.586: [Finalize Marking, 0.0005019 secs]
2017-12-05T19:00:31.334+0800: 8281.587: [GC ref-proc, 0.0009665 secs]
2017-12-05T19:00:31.335+0800: 8281.588: [Unloading, 0.0061329 secs],
0.0095552 secs]
 [Times: user=0.08 sys=0.00, real=0.01 secs] 
2017-12-05T19:00:31.344+0800: 8281.596: [GC cleanup 13G->4938M(16G),
0.0225853 secs]
 [Times: user=0.26 sys=0.00, real=0.02 secs] 
2017-12-05T19:00:31.367+0800: 8281.619: [GC concurrent-cleanup-start]
2017-12-05T19:00:31.375+0800: 8281.628: [GC concurrent-cleanup-end,
0.0086164 secs]
2017-12-05T19:24:35.312+0800: 9725.565: [GC pause (G1 Evacuation Pause)
(young), 0.0602876 secs]
   [Parallel Time: 57.3 ms, GC Workers: 13]
      [GC Worker Start (ms): Min: 9725566.0, Avg: 9725566.2, Max: 9725566.4,
Diff: 0.3]
      [Ext Root Scanning (ms): Min: 0.5, Avg: 0.7, Max: 0.9, Diff: 0.4, Sum:
8.6]
      [Update RS (ms): Min: 44.0, Avg: 44.7, Max: 46.1, Diff: 2.1, Sum:
580.6]
         [Processed Buffers: Min: 39, Avg: 75.1, Max: 97, Diff: 58, Sum:
976]
      [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.2]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0,
Sum: 0.0]
      [Object Copy (ms): Min: 9.9, Avg: 11.4, Max: 12.0, Diff: 2.1, Sum:
148.3]
      [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
         [Termination Attempts: Min: 1, Avg: 3.3, Max: 7, Diff: 6, Sum: 43]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum:
0.6]
      [GC Worker Total (ms): Min: 56.6, Avg: 56.8, Max: 57.0, Diff: 0.4,
Sum: 738.3]
      [GC Worker End (ms): Min: 9725623.0, Avg: 9725623.0, Max: 9725623.0,
Diff: 0.1]
   [Code Root Fixup: 0.1 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.3 ms]
   [Other: 2.6 ms]
      [Choose CSet: 0.0 ms]
      [Ref Proc: 0.3 ms]
      [Ref Enq: 0.0 ms]
      [Redirty Cards: 0.4 ms]
      [Humongous Register: 0.1 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 0.2 ms]
   [Eden: 472.0M(472.0M)->0.0B(456.0M) Survivors: 40.0M->56.0M Heap:
5404.0M(16.0G)->4952.0M(16.0G)]
 [Times: user=0.74 sys=0.00, real=0.06 secs] 

>From the above logs, we could find that GC on 3 nodes haven't worked one by
one. why?

My GC configuration in ignite.sh(JDK 8) is:
JVM_OPTS="-Xms16g -Xmx16g -server -XX:+AggressiveOpts -XX:MaxPermSize=256m"

JVM_OPTS="$JVM_OPTS -XX:NewSize=512m -XX:SurvivorRatio=6 -XX:+AlwaysPreTouch
-XX:+UseG1GC -XX:MaxGCPauseMillis=2000 -XX:GCTimeRatio=4
-XX:InitiatingHeapOccupancyPercent=30 -XX:G1HeapRegionSize=8M
-XX:ConcGCThreads=4 -XX:G1HeapWastePercent=10 -XX:+UseTLAB
-XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC

Do I need to improve it  following this
post:https://stackoverflow.com/questions/30796111/g1-young-gc-does-not-free-memory-to-space-exhausted?

Besides, I am not sure whether those codes consuming kafka messages are
correct or not, like:
    public static Object person_object = new Object();
    synchronized(person_object){
           personCache.put(personKey, person);      
    }
  
Does this lock affect those cache opertators on server nodes?

Thanks.
 







--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Mime
View raw message