ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nikolai Tikhonov <ntikho...@apache.org>
Subject Re: cluster hanged when client node put data into the caches
Date Tue, 12 Dec 2017 14:45:17 GMT
Hello,

You don't have to use synchronization for Ignite Cache API. This instance
already is threadsafe. I recommend to read the following documentation
page: https://apacheignite.readme.io/docs/jvm-and-system-tuning

On Sun, Dec 10, 2017 at 4:49 PM, Aurora <2565003241@qq.com> wrote:

> @Nikolai
> Your suggestion is right.
> "to-space exhausted" occured  in GC Logs on 3 out of 6 server nodes, as:
> Node 1:
> 2017-12-05T18:36:48.716+0800: 6870.705: [GC pause (G1 Evacuation Pause)
> (young), 0.4171797 secs]
>    [Parallel Time: 412.1 ms, GC Workers: 13]
>       [GC Worker Start (ms): Min: 6870705.5, Avg: 6870705.7, Max:
> 6870705.8,
> Diff: 0.3]
>       [Ext Root Scanning (ms): Min: 0.6, Avg: 0.7, Max: 0.9, Diff: 0.4,
> Sum:
> 9.6]
>       [Update RS (ms): Min: 56.8, Avg: 58.6, Max: 60.2, Diff: 3.4, Sum:
> 761.8]
>          [Processed Buffers: Min: 69, Avg: 76.7, Max: 91, Diff: 22, Sum:
> 997]
>       [Scan RS (ms): Min: 1.4, Avg: 2.8, Max: 4.7, Diff: 3.3, Sum: 36.7]
>       [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1,
> Sum: 0.2]
>       [Object Copy (ms): Min: 349.1, Avg: 349.2, Max: 349.3, Diff: 0.2,
> Sum:
> 4540.2]
>       [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.5]
>          [Termination Attempts: Min: 1, Avg: 40.3, Max: 75, Diff: 74, Sum:
> 524]
>       [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.2, Sum:
> 1.7]
>       [GC Worker Total (ms): Min: 411.4, Avg: 411.6, Max: 411.8, Diff: 0.4,
> Sum: 5350.7]
>       [GC Worker End (ms): Min: 6871117.2, Avg: 6871117.3, Max: 6871117.4,
> Diff: 0.2]
>    [Code Root Fixup: 0.1 ms]
>    [Code Root Purge: 0.0 ms]
>    [Clear CT: 0.9 ms]
>    [Other: 4.1 ms]
>       [Choose CSet: 0.0 ms]
>       [Ref Proc: 0.4 ms]
>       [Ref Enq: 0.0 ms]
>       [Redirty Cards: 0.9 ms]
>       [Humongous Register: 0.1 ms]
>       [Humongous Reclaim: 0.0 ms]
>       [Free CSet: 1.9 ms]
>    [Eden: 8280.0M(8280.0M)->0.0B(8032.0M) Survivors: 968.0M->968.0M Heap:
> 13.4G(16.0G)->5680.8M(16.0G)]
>  [Times: user=5.37 sys=0.00, real=0.42 secs]
> 2017-12-05T18:40:45.218+0800: 7107.207: [GC pause (G1 Evacuation Pause)
> (young) (to-space exhausted), 19.5379052 secs]
>    [Parallel Time: 16666.0 ms, GC Workers: 13]
>       [GC Worker Start (ms): Min: 7107207.2, Avg: 7107207.4, Max:
> 7107207.5,
> Diff: 0.3]
>       [Ext Root Scanning (ms): Min: 0.6, Avg: 0.7, Max: 0.9, Diff: 0.3,
> Sum:
> 9.1]
>       [Update RS (ms): Min: 143.9, Avg: 144.0, Max: 144.1, Diff: 0.2, Sum:
> 1871.6]
>          [Processed Buffers: Min: 78, Avg: 87.2, Max: 108, Diff: 30, Sum:
> 1134]
>       [Scan RS (ms): Min: 6.1, Avg: 6.2, Max: 6.3, Diff: 0.2, Sum: 80.3]
>       [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0,
> Sum: 0.2]
>       [Object Copy (ms): Min: 16474.9, Avg: 16481.7, Max: 16514.7, Diff:
> 39.8, Sum: 214262.3]
>       [Termination (ms): Min: 0.0, Avg: 32.9, Max: 39.6, Diff: 39.6, Sum:
> 427.8]
>          [Termination Attempts: Min: 1, Avg: 1.5, Max: 4, Diff: 3, Sum: 19]
>       [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum:
> 0.9]
>       [GC Worker Total (ms): Min: 16665.4, Avg: 16665.6, Max: 16665.7,
> Diff:
> 0.3, Sum: 216652.3]
>       [GC Worker End (ms): Min: 7123872.9, Avg: 7123872.9, Max: 7123873.0,
> Diff: 0.1]
>    [Code Root Fixup: 0.1 ms]
>    [Code Root Purge: 0.0 ms]
>    [Clear CT: 0.8 ms]
>    [Other: 2871.0 ms]
>       [Evacuation Failure: 2861.6 ms]
>       [Choose CSet: 0.0 ms]
>       [Ref Proc: 0.4 ms]
>       [Ref Enq: 0.0 ms]
>       [Redirty Cards: 6.9 ms]
>       [Humongous Register: 0.1 ms]
>       [Humongous Reclaim: 0.0 ms]
>       [Free CSet: 1.3 ms]
>    [Eden: 8032.0M(8032.0M)->0.0B(328.0M) Survivors: 968.0M->1504.0M Heap:
> 13.5G(16.0G)->12.5G(16.0G)]
>  [Times: user=43.38 sys=7.36, real=19.53 secs]
> 2017-12-05T18:41:05.194+0800: 7127.183: [GC pause (G1 Evacuation Pause)
> (young) (initial-mark), 0.9987695 secs]
>    [Parallel Time: 990.4 ms, GC Workers: 13]
>       [GC Worker Start (ms): Min: 7127183.8, Avg: 7127183.9, Max:
> 7127184.0,
> Diff: 0.2]
>       [Ext Root Scanning (ms): Min: 0.9, Avg: 1.1, Max: 1.3, Diff: 0.4,
> Sum:
> 14.3]
>       [Update RS (ms): Min: 563.4, Avg: 563.6, Max: 564.4, Diff: 1.0, Sum:
> 7327.2]
>          [Processed Buffers: Min: 1025, Avg: 1339.8, Max: 1575, Diff: 550,
> Sum: 17418]
>       [Scan RS (ms): Min: 15.0, Avg: 15.7, Max: 16.0, Diff: 1.0, Sum:
> 204.6]
>       [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0,
> Sum: 0.1]
>       [Object Copy (ms): Min: 409.2, Avg: 409.4, Max: 409.4, Diff: 0.2,
> Sum:
> 5321.6]
>       [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.6]
>          [Termination Attempts: Min: 1, Avg: 33.3, Max: 48, Diff: 47, Sum:
> 433]
>       [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum:
> 1.2]
>       [GC Worker Total (ms): Min: 989.8, Avg: 990.0, Max: 990.2, Diff: 0.4,
> Sum: 12869.6]
>       [GC Worker End (ms): Min: 7128173.8, Avg: 7128173.9, Max: 7128174.0,
> Diff: 0.2]
>    [Code Root Fixup: 0.1 ms]
>    [Code Root Purge: 0.0 ms]
>    [Clear CT: 0.3 ms]
>    [Other: 7.9 ms]
>       [Choose CSet: 0.0 ms]
>       [Ref Proc: 0.4 ms]
>       [Ref Enq: 0.0 ms]
>       [Redirty Cards: 4.8 ms]
>       [Humongous Register: 0.0 ms]
>       [Humongous Reclaim: 0.0 ms]
>       [Free CSet: 0.8 ms]
>    [Eden: 328.0M(328.0M)->0.0B(336.0M) Survivors: 1504.0M->280.0M Heap:
> 12.9G(16.0G)->12.8G(16.0G)]
>  [Times: user=12.93 sys=0.00, real=1.00 secs]
>
> Node 2:
> 2017-12-05T18:37:14.177+0800: 9497.100: [GC pause (G1 Evacuation Pause)
> (young), 0.2761496 secs]
>    [Parallel Time: 271.9 ms, GC Workers: 13]
>       [GC Worker Start (ms): Min: 9497100.0, Avg: 9497100.2, Max:
> 9497100.4,
> Diff: 0.3]
>       [Ext Root Scanning (ms): Min: 0.6, Avg: 0.8, Max: 0.9, Diff: 0.4,
> Sum:
> 10.0]
>       [Update RS (ms): Min: 19.0, Avg: 20.3, Max: 22.0, Diff: 3.0, Sum:
> 264.3]
>          [Processed Buffers: Min: 6, Avg: 13.7, Max: 42, Diff: 36, Sum:
> 178]
>       [Scan RS (ms): Min: 0.1, Avg: 0.3, Max: 0.7, Diff: 0.7, Sum: 3.5]
>       [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1,
> Sum: 0.2]
>       [Object Copy (ms): Min: 247.7, Avg: 249.3, Max: 250.5, Diff: 2.7,
> Sum:
> 3241.1]
>       [Termination (ms): Min: 0.0, Avg: 0.5, Max: 0.7, Diff: 0.7, Sum: 6.9]
>          [Termination Attempts: Min: 1, Avg: 374.3, Max: 476, Diff: 475,
> Sum: 4866]
>       [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.3, Sum:
> 2.1]
>       [GC Worker Total (ms): Min: 271.1, Avg: 271.4, Max: 271.7, Diff: 0.6,
> Sum: 3528.1]
>       [GC Worker End (ms): Min: 9497371.5, Avg: 9497371.6, Max: 9497371.7,
> Diff: 0.3]
>    [Code Root Fixup: 0.1 ms]
>    [Code Root Purge: 0.0 ms]
>    [Clear CT: 0.9 ms]
>    [Other: 3.2 ms]
>       [Choose CSet: 0.0 ms]
>       [Ref Proc: 0.3 ms]
>       [Ref Enq: 0.0 ms]
>       [Redirty Cards: 0.4 ms]
>       [Humongous Register: 0.1 ms]
>       [Humongous Reclaim: 0.0 ms]
>       [Free CSet: 2.0 ms]
>    [Eden: 9392.0M(9392.0M)->0.0B(9080.0M) Survivors: 432.0M->744.0M Heap:
> 13.5G(16.0G)->4752.0M(16.0G)]
>  [Times: user=3.54 sys=0.01, real=0.28 secs]
> 2017-12-05T18:43:29.983+0800: 9872.905: [GC pause (G1 Evacuation Pause)
> (young) (to-space exhausted), 45.1101748 secs]
>    [Parallel Time: 39495.6 ms, GC Workers: 13]
>       [GC Worker Start (ms): Min: 9872905.5, Avg: 9872905.7, Max:
> 9872905.8,
> Diff: 0.3]
>       [Ext Root Scanning (ms): Min: 0.6, Avg: 0.8, Max: 0.9, Diff: 0.3,
> Sum:
> 9.8]
>       [Update RS (ms): Min: 1180.8, Avg: 1183.2, Max: 1184.5, Diff: 3.7,
> Sum: 15382.2]
>          [Processed Buffers: Min: 63, Avg: 73.8, Max: 95, Diff: 32, Sum:
> 960]
>       [Scan RS (ms): Min: 0.1, Avg: 0.2, Max: 0.4, Diff: 0.4, Sum: 2.8]
>       [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0,
> Sum: 0.2]
>       [Object Copy (ms): Min: 38287.7, Avg: 38290.6, Max: 38310.0, Diff:
> 22.3, Sum: 497777.7]
>       [Termination (ms): Min: 0.0, Avg: 20.3, Max: 22.0, Diff: 22.0, Sum:
> 263.9]
>          [Termination Attempts: Min: 1, Avg: 20.8, Max: 32, Diff: 31, Sum:
> 271]
>       [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum:
> 1.3]
>       [GC Worker Total (ms): Min: 39495.0, Avg: 39495.2, Max: 39495.5,
> Diff:
> 0.4, Sum: 513437.7]
>       [GC Worker End (ms): Min: 9912400.8, Avg: 9912400.9, Max: 9912401.0,
> Diff: 0.2]
>    [Code Root Fixup: 0.1 ms]
>    [Code Root Purge: 0.0 ms]
>    [Clear CT: 0.8 ms]
>    [Other: 5613.6 ms]
>       [Evacuation Failure: 5603.4 ms]
>       [Choose CSet: 0.0 ms]
>       [Ref Proc: 0.4 ms]
>       [Ref Enq: 0.0 ms]
>       [Redirty Cards: 8.1 ms]
>       [Humongous Register: 0.1 ms]
>       [Humongous Reclaim: 0.0 ms]
>       [Free CSet: 1.0 ms]
>    [Eden: 9080.0M(9080.0M)->0.0B(8192.0K) Survivors: 744.0M->1640.0M Heap:
> 13.6G(16.0G)->13.2G(16.0G)]
>  [Times: user=75.59 sys=19.17, real=45.10 secs]
> 2017-12-05T18:44:15.116+0800: 9918.039: [GC pause (G1 Evacuation Pause)
> (young) (initial-mark), 1.2374672 secs]
>    [Parallel Time: 1229.6 ms, GC Workers: 13]
>       [GC Worker Start (ms): Min: 9918039.4, Avg: 9918039.5, Max:
> 9918039.6,
> Diff: 0.2]
>       [Ext Root Scanning (ms): Min: 0.8, Avg: 0.9, Max: 1.2, Diff: 0.4,
> Sum:
> 12.3]
>       [Update RS (ms): Min: 855.4, Avg: 855.5, Max: 855.7, Diff: 0.4, Sum:
> 11121.4]
>          [Processed Buffers: Min: 1844, Avg: 2442.1, Max: 2840, Diff: 996,
> Sum: 31747]
>       [Scan RS (ms): Min: 13.3, Avg: 13.4, Max: 13.5, Diff: 0.3, Sum:
> 174.2]
>       [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0,
> Sum: 0.1]
>       [Object Copy (ms): Min: 359.2, Avg: 359.2, Max: 359.3, Diff: 0.1,
> Sum:
> 4669.9]
>       [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.2]
>          [Termination Attempts: Min: 1, Avg: 19.1, Max: 33, Diff: 32, Sum:
> 248]
>       [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum:
> 1.4]
>       [GC Worker Total (ms): Min: 1229.0, Avg: 1229.2, Max: 1229.4, Diff:
> 0.4, Sum: 15979.5]
>       [GC Worker End (ms): Min: 9919268.6, Avg: 9919268.7, Max: 9919268.8,
> Diff: 0.2]
>    [Code Root Fixup: 0.1 ms]
>    [Code Root Purge: 0.0 ms]
>    [Clear CT: 0.3 ms]
>    [Other: 7.4 ms]
>       [Choose CSet: 0.0 ms]
>       [Ref Proc: 0.4 ms]
>       [Ref Enq: 0.0 ms]
>       [Redirty Cards: 4.7 ms]
>       [Humongous Register: 0.0 ms]
>       [Humongous Reclaim: 0.0 ms]
>       [Free CSet: 0.8 ms]
>    [Eden: 8192.0K(8192.0K)->0.0B(456.0M) Survivors: 1640.0M->56.0M Heap:
> 13.2G(16.0G)->13.3G(16.0G)]
>  [Times: user=16.03 sys=0.00, real=1.24 secs]
> 2017-12-05T18:44:16.354+0800: 9919.276: [GC
> concurrent-root-region-scan-start]
> 2017-12-05T18:44:16.368+0800: 9919.291: [GC concurrent-root-region-scan-
> end,
> 0.0141460 secs]
> 2017-12-05T18:44:16.368+0800: 9919.291: [GC concurrent-mark-start]
> 2017-12-05T18:44:19.218+0800: 9922.141: [GC pause (G1 Evacuation Pause)
> (young), 0.1909109 secs]
>    [Parallel Time: 187.1 ms, GC Workers: 13]
>       [GC Worker Start (ms): Min: 9922141.8, Avg: 9922141.9, Max:
> 9922142.1,
> Diff: 0.3]
>       [Ext Root Scanning (ms): Min: 0.6, Avg: 0.7, Max: 0.9, Diff: 0.3,
> Sum:
> 9.6]
>       [Update RS (ms): Min: 54.0, Avg: 54.2, Max: 54.5, Diff: 0.5, Sum:
> 704.3]
>          [Processed Buffers: Min: 58, Avg: 64.4, Max: 80, Diff: 22, Sum:
> 837]
>       [Scan RS (ms): Min: 4.9, Avg: 5.2, Max: 5.4, Diff: 0.5, Sum: 68.1]
>       [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0,
> Sum: 0.0]
>       [Object Copy (ms): Min: 126.3, Avg: 126.3, Max: 126.4, Diff: 0.2,
> Sum:
> 1641.9]
>       [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 1.6]
>          [Termination Attempts: Min: 1, Avg: 77.5, Max: 97, Diff: 96, Sum:
> 1007]
>       [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum:
> 1.0]
>       [GC Worker Total (ms): Min: 186.5, Avg: 186.7, Max: 186.9, Diff: 0.4,
> Sum: 2426.6]
>       [GC Worker End (ms): Min: 9922328.6, Avg: 9922328.6, Max: 9922328.7,
> Diff: 0.1]
>    [Code Root Fixup: 0.1 ms]
>    [Code Root Purge: 0.0 ms]
>    [Clear CT: 0.3 ms]
>    [Other: 3.4 ms]
>       [Choose CSet: 0.0 ms]
>       [Ref Proc: 0.5 ms]
>       [Ref Enq: 0.0 ms]
>       [Redirty Cards: 1.2 ms]
>       [Humongous Register: 0.1 ms]
>       [Humongous Reclaim: 0.0 ms]
>       [Free CSet: 0.4 ms]
>    [Eden: 456.0M(456.0M)->0.0B(424.0M) Survivors: 56.0M->88.0M Heap:
> 13.8G(16.0G)->13.7G(16.0G)]
>  [Times: user=2.43 sys=0.01, real=0.19 secs]
>
> Node 3:
> 2017-12-05T18:41:07.782+0800: 7118.035: [GC pause (G1 Evacuation Pause)
> (young), 0.5114032 secs]
>    [Parallel Time: 506.9 ms, GC Workers: 13]
>       [GC Worker Start (ms): Min: 7118034.9, Avg: 7118035.0, Max:
> 7118035.2,
> Diff: 0.3]
>       [Ext Root Scanning (ms): Min: 0.6, Avg: 0.7, Max: 0.9, Diff: 0.3,
> Sum:
> 9.3]
>       [Update RS (ms): Min: 50.6, Avg: 51.5, Max: 52.5, Diff: 1.9, Sum:
> 669.4]
>          [Processed Buffers: Min: 20, Avg: 31.3, Max: 56, Diff: 36, Sum:
> 407]
>       [Scan RS (ms): Min: 0.1, Avg: 0.2, Max: 0.6, Diff: 0.6, Sum: 3.0]
>       [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0,
> Sum: 0.2]
>       [Object Copy (ms): Min: 453.0, Avg: 453.8, Max: 454.3, Diff: 1.3,
> Sum:
> 5899.5]
>       [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.5]
>          [Termination Attempts: Min: 1, Avg: 20.4, Max: 27, Diff: 26, Sum:
> 265]
>       [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.2, Sum:
> 1.8]
>       [GC Worker Total (ms): Min: 506.2, Avg: 506.4, Max: 506.7, Diff: 0.5,
> Sum: 6583.7]
>       [GC Worker End (ms): Min: 7118541.4, Avg: 7118541.5, Max: 7118541.6,
> Diff: 0.2]
>    [Code Root Fixup: 0.1 ms]
>    [Code Root Purge: 0.0 ms]
>    [Clear CT: 0.9 ms]
>    [Other: 3.5 ms]
>       [Choose CSet: 0.0 ms]
>       [Ref Proc: 0.5 ms]
>       [Ref Enq: 0.0 ms]
>       [Redirty Cards: 0.5 ms]
>       [Humongous Register: 0.1 ms]
>       [Humongous Reclaim: 0.0 ms]
>       [Free CSet: 1.8 ms]
>    [Eden: 9208.0M(9208.0M)->0.0B(7160.0M) Survivors: 384.0M->1600.0M Heap:
> 13.5G(16.0G)->5928.0M(16.0G)]
>  [Times: user=6.59 sys=0.00, real=0.52 secs]
> 2017-12-05T18:59:32.939+0800: 8223.192: [GC pause (G1 Evacuation Pause)
> (young) (to-space exhausted), 55.7872972 secs]
>    [Parallel Time: 50147.5 ms, GC Workers: 13]
>       [GC Worker Start (ms): Min: 8223192.1, Avg: 8223192.2, Max:
> 8223192.4,
> Diff: 0.3]
>       [Ext Root Scanning (ms): Min: 0.6, Avg: 0.7, Max: 0.9, Diff: 0.3,
> Sum:
> 9.2]
>       [Update RS (ms): Min: 514.3, Avg: 516.5, Max: 523.9, Diff: 9.6, Sum:
> 6714.9]
>          [Processed Buffers: Min: 105, Avg: 122.5, Max: 165, Diff: 60, Sum:
> 1593]
>       [Scan RS (ms): Min: 1916.0, Avg: 1923.1, Max: 1925.4, Diff: 9.5, Sum:
> 25000.0]
>       [Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2,
> Sum: 1.0]
>       [Object Copy (ms): Min: 47705.9, Avg: 47706.4, Max: 47707.0, Diff:
> 1.0, Sum: 620183.4]
>       [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.2, Sum: 1.8]
>          [Termination Attempts: Min: 1, Avg: 28.2, Max: 47, Diff: 46, Sum:
> 367]
>       [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum:
> 1.0]
>       [GC Worker Total (ms): Min: 50146.9, Avg: 50147.0, Max: 50147.2,
> Diff:
> 0.3, Sum: 651911.3]
>       [GC Worker End (ms): Min: 8273339.2, Avg: 8273339.3, Max: 8273339.3,
> Diff: 0.2]
>    [Code Root Fixup: 0.1 ms]
>    [Code Root Purge: 0.0 ms]
>    [Clear CT: 0.6 ms]
>    [Other: 5639.1 ms]
>       [Evacuation Failure: 5622.5 ms]
>       [Choose CSet: 0.0 ms]
>       [Ref Proc: 0.3 ms]
>       [Ref Enq: 0.0 ms]
>       [Redirty Cards: 12.1 ms]
>       [Humongous Register: 0.1 ms]
>       [Humongous Reclaim: 0.0 ms]
>       [Free CSet: 3.7 ms]
>    [Eden: 7160.0M(7160.0M)->0.0B(8192.0K) Survivors: 1600.0M->1232.0M
> Heap:
> 13.1G(16.0G)->13.8G(16.0G)]
>  [Times: user=96.95 sys=24.80, real=55.78 secs]
> 2017-12-05T19:00:28.876+0800: 8279.128: [GC pause (G1 Evacuation Pause)
> (young) (initial-mark), 2.3525928 secs]
>    [Parallel Time: 2345.5 ms, GC Workers: 13]
>       [GC Worker Start (ms): Min: 8279129.3, Avg: 8279129.4, Max:
> 8279129.5,
> Diff: 0.2]
>       [Ext Root Scanning (ms): Min: 0.7, Avg: 0.8, Max: 1.0, Diff: 0.3,
> Sum:
> 10.9]
>       [Update RS (ms): Min: 1933.1, Avg: 1933.3, Max: 1933.6, Diff: 0.5,
> Sum: 25132.7]
>          [Processed Buffers: Min: 2505, Avg: 3215.6, Max: 3919, Diff: 1414,
> Sum: 41803]
>       [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.5]
>       [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1,
> Sum: 0.1]
>       [Object Copy (ms): Min: 410.4, Avg: 410.7, Max: 410.9, Diff: 0.5,
> Sum:
> 5338.6]
>       [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.7]
>          [Termination Attempts: Min: 1, Avg: 84.6, Max: 120, Diff: 119,
> Sum:
> 1100]
>       [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum:
> 1.2]
>       [GC Worker Total (ms): Min: 2344.9, Avg: 2345.1, Max: 2345.2, Diff:
> 0.3, Sum: 30485.8]
>       [GC Worker End (ms): Min: 8281474.4, Avg: 8281474.5, Max: 8281474.6,
> Diff: 0.2]
>    [Code Root Fixup: 0.1 ms]
>    [Code Root Purge: 0.0 ms]
>    [Clear CT: 0.4 ms]
>    [Other: 6.6 ms]
>       [Choose CSet: 0.0 ms]
>       [Ref Proc: 0.7 ms]
>       [Ref Enq: 0.0 ms]
>       [Redirty Cards: 3.5 ms]
>       [Humongous Register: 0.0 ms]
>       [Humongous Reclaim: 0.0 ms]
>       [Free CSet: 0.4 ms]
>    [Eden: 8192.0K(8192.0K)->0.0B(472.0M) Survivors: 1232.0M->40.0M Heap:
> 13.8G(16.0G)->13.9G(16.0G)]
>  [Times: user=30.54 sys=0.00, real=2.35 secs]
> 2017-12-05T19:00:31.229+0800: 8281.481: [GC
> concurrent-root-region-scan-start]
> 2017-12-05T19:00:31.230+0800: 8281.482: [GC concurrent-root-region-scan-
> end,
> 0.0008827 secs]
> 2017-12-05T19:00:31.230+0800: 8281.482: [GC concurrent-mark-start]
> 2017-12-05T19:00:31.332+0800: 8281.585: [GC concurrent-mark-end, 0.1025943
> secs]
> 2017-12-05T19:00:31.334+0800: 8281.586: [GC remark
> 2017-12-05T19:00:31.334+0800: 8281.586: [Finalize Marking, 0.0005019 secs]
> 2017-12-05T19:00:31.334+0800: 8281.587: [GC ref-proc, 0.0009665 secs]
> 2017-12-05T19:00:31.335+0800: 8281.588: [Unloading, 0.0061329 secs],
> 0.0095552 secs]
>  [Times: user=0.08 sys=0.00, real=0.01 secs]
> 2017-12-05T19:00:31.344+0800: 8281.596: [GC cleanup 13G->4938M(16G),
> 0.0225853 secs]
>  [Times: user=0.26 sys=0.00, real=0.02 secs]
> 2017-12-05T19:00:31.367+0800: 8281.619: [GC concurrent-cleanup-start]
> 2017-12-05T19:00:31.375+0800: 8281.628: [GC concurrent-cleanup-end,
> 0.0086164 secs]
> 2017-12-05T19:24:35.312+0800: 9725.565: [GC pause (G1 Evacuation Pause)
> (young), 0.0602876 secs]
>    [Parallel Time: 57.3 ms, GC Workers: 13]
>       [GC Worker Start (ms): Min: 9725566.0, Avg: 9725566.2, Max:
> 9725566.4,
> Diff: 0.3]
>       [Ext Root Scanning (ms): Min: 0.5, Avg: 0.7, Max: 0.9, Diff: 0.4,
> Sum:
> 8.6]
>       [Update RS (ms): Min: 44.0, Avg: 44.7, Max: 46.1, Diff: 2.1, Sum:
> 580.6]
>          [Processed Buffers: Min: 39, Avg: 75.1, Max: 97, Diff: 58, Sum:
> 976]
>       [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.2]
>       [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0,
> Sum: 0.0]
>       [Object Copy (ms): Min: 9.9, Avg: 11.4, Max: 12.0, Diff: 2.1, Sum:
> 148.3]
>       [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
>          [Termination Attempts: Min: 1, Avg: 3.3, Max: 7, Diff: 6, Sum: 43]
>       [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum:
> 0.6]
>       [GC Worker Total (ms): Min: 56.6, Avg: 56.8, Max: 57.0, Diff: 0.4,
> Sum: 738.3]
>       [GC Worker End (ms): Min: 9725623.0, Avg: 9725623.0, Max: 9725623.0,
> Diff: 0.1]
>    [Code Root Fixup: 0.1 ms]
>    [Code Root Purge: 0.0 ms]
>    [Clear CT: 0.3 ms]
>    [Other: 2.6 ms]
>       [Choose CSet: 0.0 ms]
>       [Ref Proc: 0.3 ms]
>       [Ref Enq: 0.0 ms]
>       [Redirty Cards: 0.4 ms]
>       [Humongous Register: 0.1 ms]
>       [Humongous Reclaim: 0.0 ms]
>       [Free CSet: 0.2 ms]
>    [Eden: 472.0M(472.0M)->0.0B(456.0M) Survivors: 40.0M->56.0M Heap:
> 5404.0M(16.0G)->4952.0M(16.0G)]
>  [Times: user=0.74 sys=0.00, real=0.06 secs]
>
> From the above logs, we could find that GC on 3 nodes haven't worked one by
> one. why?
>
> My GC configuration in ignite.sh(JDK 8) is:
> JVM_OPTS="-Xms16g -Xmx16g -server -XX:+AggressiveOpts -XX:MaxPermSize=256m"
>
> JVM_OPTS="$JVM_OPTS -XX:NewSize=512m -XX:SurvivorRatio=6
> -XX:+AlwaysPreTouch
> -XX:+UseG1GC -XX:MaxGCPauseMillis=2000 -XX:GCTimeRatio=4
> -XX:InitiatingHeapOccupancyPercent=30 -XX:G1HeapRegionSize=8M
> -XX:ConcGCThreads=4 -XX:G1HeapWastePercent=10 -XX:+UseTLAB
> -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
>
> Do I need to improve it  following this
> post:https://stackoverflow.com/questions/30796111/g1-
> young-gc-does-not-free-memory-to-space-exhausted?
>
> Besides, I am not sure whether those codes consuming kafka messages are
> correct or not, like:
>     public static Object person_object = new Object();
>     synchronized(person_object){
>            personCache.put(personKey, person);
>     }
>
> Does this lock affect those cache opertators on server nodes?
>
> Thanks.
>
>
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Mime
View raw message