cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anthony Grasso <anthony.gra...@gmail.com>
Subject Re: [EXTERNAL] Cassandra cluster add new node slowly
Date Thu, 04 Jan 2018 06:00:50 GMT
The speed at which compactions operate is also physically restricted by the
speed of the disk. If the disks used on the new node are HDDs, then
increasing the compaction throughput will be of little help. However, if
the disks on the new node are SSDs then increasing the compaction
throughput to at least 64MB/s should help speed up compactions.

Regards,
Anthony

On 4 January 2018 at 14:13, qf zhou <zhouqf2013@gmail.com> wrote:

> The cassandra version is 3.0.9.
>
> I  have changed the heap size (about  32G). Also, the streaming
> throughput is set 800MB/sec,  and the streaming_socket_timeout_in_ms is
> default 86400000.
> I suspect  the  compactionthroughput has an influence on the new node
> joining.   The command  nodetool | getcompactionthroughput says  'Current
> compaction throughput: 32 MB/s’.
>
>
>
>
> 在 2018年1月4日,上午4:59,Durity, Sean R <SEAN_R_DURITY@homedepot.com>
写道:
>
> You don’t mention the version, but here are some general suggestions
>
> -          2 GB heap is very small for a node, especially with 1 TB+ of
> data. What is the physical RAM on the host? In general, you want ½ of
> physical RAM for the JVM. (Look in jvm.options or cassandra-env.sh)
> -          You can change the streaming throughput from the existing
> nodes, if it looks like the new node can handle it. Look at nodetool
> setstreamthroughput. Default is 200 (MB/sec).
> -          You might want to check for a streaming_socket_timeout_in_ms.
> This has changed over the versions. Some details are at:
> https://issues.apache.org/jira/browse/CASSANDRA-11839. 24 hours is good
> recommendation.
> -          If your new node can’t compact fast enough to keep disk usage
> down, look at compactionthroughput on that node
> -          nodetool netstats | grep –v “100%” is a good way to see what
> is happening/if anything is stuck. Newer versions give a bit more info on
> progress.
> -          Don’t forget to run cleanup on existing nodes after the new
> nodes are added.
>
>
>
> Sean Durity
> *From:* qf zhou [mailto:zhouqf2013@gmail.com <zhouqf2013@gmail.com>]
> *Sent:* Tuesday, January 02, 2018 10:30 PM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Cassandra cluster add new node slowly
>
> The cluster has  3 nodes,  and  the data in each node is  about 1.2 T.  I
> want to add two new nodes to expand the cluster.
>
> Following the instructions from the datastax  website, ie,  (
> http://docs.datastax.com/en/archived/cassandra/3.x/cassandra/operations/
> opsAddNodeToCluster.html
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.datastax.com_en_archived_cassandra_3.x_cassandra_operations_opsAddNodeToCluster.html&d=DwMFoQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=AfP0BxfTG1o_-2nRtGSjngs45YndH060h39eVlypVso&s=AaiOypurMtWA470038fHMbaXJPIZeYtLzv3Yt9vf1Sc&e=>),
>
>
> I try to add one  node  to  the cluster.  However,  it  is  too slow  and
> time cost too  much.  After about  24 hours,  it still didn’t  success.
>
> I run the command: nodetool netstats  on the new node,  it  shows that:
>
> (tb1fullwithstate2  is a big table and 90% of  the cluster data  is  in
> it.   Here I use CompactionStrategy: TimeWindowCompactionStrategy).
>
> /*.*.*.3
>         Receiving 136 files, 328573794609 bytes total. Already received 8
> files, 5774621188 bytes total
>             tb1/tb1fullwithneweststatetest 3758271/3758271 bytes(100%)
> received from idx:0/*.*.*.3
>             system_distributed/repair_history 57534/57534 bytes(100%)
> received from idx:0/*.*.*.3
>             system_distributed/parent_repair_history 507660/507660
> bytes(100%) received from idx:0/*.*.*.3
>             tb1/tb1_device_last_state_eachday 15754096/15754096
> bytes(100%) received from idx:0/*.*.*.3
>             mytest1/tb1_test1 8143775/8143775 bytes(100%) received from
> idx:0/*.*.*.3
>             tb1/tb1fullwithstate 2251191007/2251191007 bytes(100%)
> received from idx:0/*.*.*.3
>             applocationinfo/weiyirong_app 2760/2760 bytes(100%) received
> from idx:0/*.*.*.3
>             tb1/tb1fullwithstate2 3490748006/4909554503 bytes(71%)
> received from idx:0/*.*.*.3
>             tb1/tb1fullwithneweststate 4458079/4458079 bytes(100%)
> received from idx:0/*.*.*.3
>     /*.*.*.2
>         Receiving 136 files, 336762487360 bytes total. Already received 3
> files, 5695770181 bytes total
>             system_distributed/repair_history 31684/31684 bytes(100%)
> received from idx:0/*.*.*.2
>             tb1/tb1fullwithstate 908260516/908260516 bytes(100%) received
> from idx:0/*.*.*.2
>             tb1/tb1fullwithstate2 4783622958/4990450588 bytes(95%)
> received from idx:0/*.*.*.2
>             tb1/tb1fullwithneweststate 3855023/3855023 bytes(100%)
> received from idx:0/*.*.*.2
>     /*.*.*.4
>         Receiving 132 files, 236250553620 bytes total. Already received 10
> files, 3117465128 bytes total
>             mytest1/wordstest2 46/46 bytes(100%) received from
> idx:0/*.*.*.4
>             tb1/tb1fullwithneweststatetest 3416891/3416891 bytes(100%)
> received from idx:0/*.*.*.4
>             system_distributed/repair_history 39720/39720 bytes(100%)
> received from idx:0/*.*.*.4
>             system_distributed/parent_repair_history 452250/452250
> bytes(100%) received from idx:0/*.*.*.4
>             mytest1/weblogs 104/104 bytes(100%) received from idx:0/*.*.*.4
>             tb1/tb1_device_last_state_eachday 12670998/12670998
> bytes(100%) received from idx:0/*.*.*.4
>             mytest1/tb1_test1 3257952/3257952 bytes(100%) received from
> idx:0/*.*.*.4
>             tb1/tb1fullwithstate 647702056/647702056 bytes(100%) received
> from idx:0/*.*.*.4
>             applocationinfo/weiyirong_app 3509/3509 bytes(100%) received
> from idx:0/*.*.*.4
>             tb1/tb1fullwithstate2 2446436305/3566562762 bytes(68%)
> received from idx:0/*.*.*.4
>             tb1/tb1fullwithneweststate 3485297/3485297 bytes(100%)
> received from idx:0/*.*.*.4
>
>
>
> check  the log in the  logs/system.log,  it shows that:
>
>
> INFO  06:09:33 Updating topology for /*.*.*.2
> INFO  06:09:33 Updating topology for /*.*.*.2
> INFO  06:09:33 Node /*.*.*.3 has restarted, now UP
> INFO  06:09:33 InetAddress /*.*.*.3 is now UP
> INFO  06:09:33 Node /*.*.*.3 state jump to NORMAL
> INFO  06:09:33 Updating topology for /*.*.*.3
> INFO  06:09:33 Updating topology for /*.*.*.3
> INFO  06:09:33 Handshaking version with /*.*.*.2
> INFO  06:09:33 Handshaking version with /*.*.*.3
> INFO  06:09:36 Handshaking version with /*.*.*.4
> INFO  06:10:03 JOINING: Starting to bootstrap...
> INFO  06:10:04 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Executing
> streaming plan for Bootstrap
> INFO  06:10:04 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Starting
> streaming to /*.*.*.4
> INFO  06:10:04 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75, ID#0]
> Beginning stream session with /*.*.*.4
> INFO  06:10:04 Deleted hint file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527790874-1.hints
> INFO  06:10:04 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527790874-1.hints to endpoint c5271e74-19a1-4cee-98d7-
> dc169cf87
> INFO  06:10:11 Deleted hint file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527800873-1.hints
> INFO  06:10:11 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527800873-1.hints to endpoint c5271e74-19a1-4cee-98d7-
> dc169cf87
> INFO  06:10:12 Deleted hint file c555e44c-9590-4f45-aea4-
> f5eca68180b2-1514527790881-1.hints
> INFO  06:10:12 Finished hinted handoff of file c555e44c-9590-4f45-aea4-
> f5eca68180b2-1514527790881-1.hints to endpoint c555e44c-9590-4f45-aea4-
> f5eca6818
> INFO  06:10:20 Deleted hint file c555e44c-9590-4f45-aea4-
> f5eca68180b2-1514527800880-1.hints
> INFO  06:10:20 Finished hinted handoff of file c555e44c-9590-4f45-aea4-
> f5eca68180b2-1514527800880-1.hints to endpoint c555e44c-9590-4f45-aea4-
> f5eca6818
> INFO  06:10:21 Deleted hint file c623bbc0-9839-4d2d-8ff3-
> db7115719d59-1514527790884-1.hints
> INFO  06:10:21 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-
> db7115719d59-1514527790884-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-
> db7115719
> INFO  06:10:28 Deleted hint file c623bbc0-9839-4d2d-8ff3-
> db7115719d59-1514527800887-1.hints
> INFO  06:10:28 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-
> db7115719d59-1514527800887-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-
> db7115719
> INFO  06:10:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75 ID#0]
> Prepare completed. Receiving 132 files(220.025GiB), sending 0
> files(0.000KiB)
> INFO  06:10:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Starting
> streaming to /*.*.*.2
> INFO  06:10:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75, ID#0]
> Beginning stream session with /*.*.*.2
> INFO  06:10:34 Deleted hint file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527810873-1.hints
> INFO  06:10:34 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527810873-1.hints to endpoint c5271e74-19a1-4cee-98d7-
> dc169cf87
> INFO  06:10:42 Deleted hint file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527820873-1.hints
> INFO  06:10:42 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527820873-1.hints to endpoint c5271e74-19a1-4cee-98d7-
> dc169cf87
> INFO  06:10:58 Deleted hint file c555e44c-9590-4f45-aea4-
> f5eca68180b2-1514527810881-1.hints
> INFO  06:10:58 Finished hinted handoff of file c555e44c-9590-4f45-aea4-
> f5eca68180b2-1514527810881-1.hints to endpoint c555e44c-9590-4f45-aea4-
> f5eca6818
> INFO  06:11:03 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75 ID#0]
> Prepare completed. Receiving 136 files(313.635GiB), sending 0
> files(0.000KiB)
> INFO  06:11:03 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Starting
> streaming to /*.*.*.3
> INFO  06:11:03 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75, ID#0]
> Beginning stream session with /*.*.*.3
> INFO  06:11:06 Deleted hint file c555e44c-9590-4f45-aea4-
> f5eca68180b2-1514527830880-1.hints
> INFO  06:11:06 Finished hinted handoff of file c555e44c-9590-4f45-aea4-
> f5eca68180b2-1514527830880-1.hints to endpoint c555e44c-9590-4f45-aea4-
> f5eca6818
> INFO  06:11:29 Deleted hint file c623bbc0-9839-4d2d-8ff3-
> db7115719d59-1514527810889-1.hints
> INFO  06:11:29 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-
> db7115719d59-1514527810889-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-
> db7115719
> INFO  06:11:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75 ID#0]
> Prepare completed. Receiving 136 files(306.008GiB), sending 0
> files(0.000KiB)
> INFO  06:11:36 Deleted hint file c623bbc0-9839-4d2d-8ff3-
> db7115719d59-1514527840887-1.hints
> INFO  06:11:36 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-
> db7115719d59-1514527840887-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-
> db7115719
> INFO  06:11:49 Deleted hint file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527830873-1.hints
> INFO  06:11:49 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527830873-1.hints to endpoint c5271e74-19a1-4cee-98d7-
> dc169cf87
> INFO  06:11:56 Deleted hint file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527850874-1.hints
> INFO  06:11:56 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527850874-1.hints to endpoint c5271e74-19a1-4cee-98d7-
> dc169cf87
> INFO  06:12:26 Deleted hint file c555e44c-9590-4f45-aea4-
> f5eca68180b2-1514527840880-1.hints
> INFO  06:12:26 Finished hinted handoff of file c555e44c-9590-4f45-aea4-
> f5eca68180b2-1514527840880-1.hints to endpoint c555e44c-9590-4f45-aea4-
> f5eca6818
> INFO  06:12:45 Deleted hint file c623bbc0-9839-4d2d-8ff3-
> db7115719d59-1514527850883-1.hints
> INFO  06:12:45 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-
> db7115719d59-1514527850883-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-
> db7115719
> INFO  06:12:56 Deleted hint file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527860874-1.hints
> INFO  06:12:56 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527860874-1.hints to endpoint c5271e74-19a1-4cee-98d7-
> dc169cf87
> INFO  06:24:03 G1 Young Generation GC in 242ms.  G1 Eden Space:
> 18924699648 -> 0; G1 Old Gen: 4360876536 -> 5095617016; G1 Survivor Space:
> 1677721600 -
> INFO  06:26:06 G1 Young Generation GC in 215ms.  G1 Eden Space:
> 18220056576 -> 0; G1 Old Gen: 5981077496 -> 6597610488; G1 Survivor Space:
> 1493172224 -
> INFO  06:29:04 G1 Young Generation GC in 225ms.  G1 Eden Space:
> 18538823680 -> 0; G1 Old Gen: 7910457840 -> 8860639728; G1 Survivor Space:
> 1895825408 -
> INFO  06:33:22 G1 Young Generation GC in 214ms.  G1 Eden Space:
> 16559112192 -> 0; G1 Old Gen: 11868321784 -> 11941600752; G1 Survivor
> Space: 1241513984
> INFO  06:33:33 G1 Young Generation GC in 205ms.  G1 Eden Space: 8069840896
> -> 0; G1 Old Gen: 11941600752 -> 13683606520; G1 Survivor Space: 1895825408
> INFO  06:41:01 G1 Young Generation GC in 383ms.  G1 Eden Space:
> 18874368000 -> 0; G1 Old Gen: 4770399736 -> 5551887344; G1 Survivor Space:
> 1728053248 -
> INFO  06:45:07 G1 Young Generation GC in 226ms.  G1 Eden Space:
> 19025362944 -> 0; G1 Old Gen: 8036286448 -> 8739698672; G1 Survivor Space:
> 1577058304 -
> INFO  06:47:11 G1 Young Generation GC in 203ms.  G1 Eden Space:
> 18538823680 -> 0; G1 Survivor Space: 1191182336 -> 1778384896;
> INFO  06:47:22 G1 Young Generation GC in 241ms.  G1 Eden Space: 8086618112
> -> 0; G1 Old Gen: 9999220720 -> 11591876088; G1 Survivor Space: 1778384896 -
> INFO  06:49:43 G1 Young Generation GC in 201ms.  G1 Eden Space:
> 16022241280 -> 0; G1 Old Gen: 11591876088 -> 12084766712; G1 Survivor
> Space: 1644167168
> INFO  06:53:21 G1 Young Generation GC in 322ms.  G1 Eden Space:
> 13237223424 -> 0; G1 Old Gen: 14755561464 -> 15445886448; G1 Survivor
> Space: 1459617792
> INFO  06:55:34 G1 Young Generation GC in 343ms.  G1 Eden Space:
> 16525557760 -> 0; G1 Old Gen: 2939907056 -> 3636770792; G1 Survivor Space:
> 1610612736 -
> INFO  07:09:30 Redistributing index summaries
> INFO  07:14:25 G1 Young Generation GC in 270ms.  G1 Eden Space:
> 15602810880 -> 0; G1 Old Gen: 4360518136 -> 5243274224; G1 Survivor Space:
> 1610612736 -
> INFO  07:16:51 G1 Young Generation GC in 229ms.  G1 Eden Space:
> 19008585728 -> 0; G1 Old Gen: 6132072440 -> 6815145976; G1 Survivor Space:
> 1593835520 -
> INFO  07:19:34 G1 Young Generation GC in 215ms.  G1 Eden Space:
> 18991808512 -> 0; G1 Old Gen: 8689337336 -> 9071161328; G1 Survivor Space:
> 1610612736 -
> INFO  07:23:35 G1 Young Generation GC in 293ms.  G1 Eden Space:
> 15753805824 -> 0; G1 Old Gen: 11777605616 -> 12289649640; G1 Survivor
> Space: 1694498816
> INFO  07:29:42 G1 Young Generation GC in 362ms.  G1 Eden Space:
> 15955132416 -> 0; G1 Old Gen: 3742637560 -> 4410361824; G1 Survivor Space:
> 1644167168 -
> INFO  07:35:12 G1 Young Generation GC in 208ms.  G1 Eden Space:
> 19478347776 -> 0; G1 Old Gen: 7199737832 -> 7208126432; G1 Survivor Space:
> 1124073472 -
> INFO  07:35:33 G1 Young Generation GC in 219ms.  G1 Eden Space:
> 18740150272 -> 0; G1 Old Gen: 7208126432 -> 8072478184; G1 Survivor Space:
> 1862270976 -
> INFO  07:45:19 G1 Young Generation GC in 213ms.  G1 Eden Space:
> 13153337344 -> 0; G1 Old Gen: 15418261472 -> 16123805152; G1 Survivor
> Space: 1174405120
> INFO  07:48:08 G1 Young Generation GC in 335ms.  G1 Eden Space:
> 18387828736 -> 0; G1 Old Gen: 3852453368 -> 4464797168; G1 Survivor Space:
> 1644167168 -
> INFO  07:48:08 G1 Young Generation GC in 209ms.  G1 Eden Space: 352321536
> -> 0; G1 Old Gen: 4464797168 -> 5989466096; G1 Survivor Space: 1476395008
> ->
> INFO  07:50:25 G1 Young Generation GC in 221ms.  G1 Eden Space:
> 17498636288 -> 0; G1 Old Gen: 5981077496 -> 6488668656; G1 Survivor Space:
> 1593835520 -
> INFO  08:03:31 G1 Young Generation GC in 748ms.  G1 Eden Space: 8673820672
> -> 0; G1 Old Gen: 4386367472 -> 5874747352; G1 Survivor Space: 1627389952 ->
> INFO  08:09:30 Redistributing index summaries
> INFO  08:10:05 G1 Young Generation GC in 213ms.  G1 Eden Space:
> 18220056576 -> 0; G1 Old Gen: 9276024272 -> 9969348048; G1 Survivor Space:
> 1711276032 -
> INFO  08:18:59 G1 Young Generation GC in 228ms.  G1 Eden Space:
> 17347641344 -> 0; G1 Old Gen: 4546626040 -> 5180788216; G1 Survivor Space:
> 1593835520 -
> INFO  08:33:12 G1 Young Generation GC in 207ms.  G1 Eden Space:
> 19411238912 -> 0; G1 Survivor Space: 1191182336 -> 1828716544;
> INFO  08:33:13 G1 Young Generation GC in 273ms.  G1 Eden Space: 452984832
> -> 0; G1 Old Gen: 4668340728 -> 6320276984; G1 Survivor Space: 1828716544
> ->
> INFO  08:41:59 G1 Young Generation GC in 202ms.  G1 Eden Space:
> 19025362944 -> 0; G1 Old Gen: 8701765104 -> 8927144944; G1 Survivor Space:
> 1577058304 -
> INFO  08:56:31 G1 Young Generation GC in 205ms.  G1 Eden Space:
> 11072962560 -> 0; G1 Old Gen: 9563013104 -> 11068430328; G1 Survivor Space:
> 1660944384
> INFO  09:05:49 G1 Young Generation GC in 367ms.  G1 Eden Space:
> 18522046464 -> 0; G1 Old Gen: 4009754608 -> 4732347872; G1 Survivor Space:
> 1778384896 -
> INFO  09:09:30 Redistributing index summaries
> INFO  09:14:52 G1 Young Generation GC in 213ms.  G1 Eden Space:
> 15267266560 -> 0; G1 Old Gen: 12155092968 -> 12906235360; G1 Survivor
> Space: 1744830464
> INFO  09:17:23 G1 Young Generation GC in 203ms.  G1 Eden Space:
> 14159970304 -> 0; G1 Old Gen: 14779568608 -> 14771180008; G1 Survivor
> Space: 989855744
> INFO  09:17:45 G1 Young Generation GC in 204ms.  G1 Eden Space:
> 12616466432 -> 0; G1 Old Gen: 14771180008 -> 15500684768; G1 Survivor
> Space: 1761607680
> INFO  09:20:52 G1 Young Generation GC in 361ms.  G1 Eden Space:
> 19025362944 -> 0; G1 Old Gen: 4351990768 -> 5086824424; G1 Survivor Space:
> 1577058304 -
> INFO  09:23:51 G1 Young Generation GC in 207ms.  G1 Eden Space:
> 18891145216 -> 0; G1 Old Gen: 7001879024 -> 7716560880; G1 Survivor Space:
> 1711276032 -
> INFO  09:29:20 G1 Young Generation GC in 202ms.  G1 Eden Space:
> 17028874240 -> 0; G1 Survivor Space: 1040187392 -> 1828716544;
> INFO  09:29:40 G1 Young Generation GC in 219ms.  G1 Eden Space:
> 15451815936 -> 0; G1 Old Gen: 11802771944 -> 12574988776; G1 Survivor
> Space: 1828716544
> INFO  09:44:51 G1 Young Generation GC in 202ms.  G1 Eden Space:
> 18001952768 -> 0; G1 Old Gen: 9325481456 -> 10085596664; G1 Survivor Space:
> 1795162112
> INFO  09:45:23 G1 Young Generation GC in 209ms.  G1 Eden Space: 9177137152
> -> 0; G1 Old Gen: 10093985264 -> 12309118960; G1 Survivor Space: 1677721600
> INFO  09:50:33 G1 Young Generation GC in 201ms.  G1 Eden Space:
> 12901679104 -> 0; G1 Old Gen: 14487498224 -> 15268560880; G1 Survivor
> Space: 1761607680
> INFO  10:08:09 Saved KeyCache (2989 items) in 92 ms
> INFO  10:09:30 Redistributing index summaries
> INFO  10:22:17 G1 Young Generation GC in 216ms.  G1 Eden Space:
> 19495124992 -> 0; G1 Old Gen: 7742571504 -> 7734182904; G1 Survivor Space:
> 1107296256 -
> INFO  10:25:31 G1 Young Generation GC in 206ms.  G1 Eden Space:
> 18622709760 -> 0; G1 Survivor Space: 1040187392 -> 1711276032;
> INFO  10:34:54 G1 Young Generation GC in 251ms.  G1 Eden Space:
> 18991808512 -> 0; G1 Old Gen: 5066719736 -> 5691775992; G1 Survivor Space:
> 1610612736 -
> INFO  10:40:08 G1 Young Generation GC in 209ms.  G1 Eden Space:
> 19025362944 -> 0; G1 Old Gen: 9000976880 -> 9090653168; G1 Survivor Space:
> 1442840576 -
> INFO  10:40:08 G1 Young Generation GC in 223ms.  G1 Eden Space: 469762048
> -> 0; G1 Old Gen: 9090653168 -> 11140124152; G1 Survivor Space: 1862270976
> ->
> INFO  10:52:15 G1 Young Generation GC in 219ms.  G1 Eden Space:
> 15820914688 -> 0; G1 Old Gen: 5331537912 -> 5828939768; G1 Survivor Space:
> 1275068416 -
> INFO  10:55:30 G1 Young Generation GC in 207ms.  G1 Eden Space:
> 15804137472 -> 0; G1 Old Gen: 7501364216 -> 8001282032; G1 Survivor Space:
> 1459617792 -
> INFO  11:07:57 G1 Young Generation GC in 295ms.  G1 Eden Space:
> 19159580672 -> 0; G1 Old Gen: 4301379064 -> 4919709168; G1 Survivor Space:
> 1442840576 -
> INFO  11:09:30 Redistributing index summaries
> INFO  11:13:06 G1 Young Generation GC in 204ms.  G1 Eden Space:
> 19126026240 -> 0; G1 Old Gen: 8044675064 -> 8659737080; G1 Survivor Space:
> 1476395008 -
> INFO  11:27:07 G1 Young Generation GC in 202ms.  G1 Eden Space:
> 19159580672 -> 0; G1 Old Gen: 4328521712 -> 4905718776; G1 Survivor Space:
> 1442840576 -
> INFO  11:30:08 G1 Young Generation GC in 209ms.  G1 Eden Space:
> 19243466752 -> 0; G1 Survivor Space: 1358954496 -> 1828716544;
> INFO  11:30:25 G1 Young Generation GC in 226ms.  G1 Eden Space:
> 14277410816 -> 0; G1 Old Gen: 6349296624 -> 7552639472; G1 Survivor Space:
> 1828716544 -
> INFO  11:32:30 G1 Young Generation GC in 215ms.  G1 Eden Space:
> 19142803456 -> 0; G1 Survivor Space: 1459617792 -> 2046820352;
> INFO  11:33:01 G1 Young Generation GC in 247ms.  G1 Eden Space:
> 17263755264 -> 0; G1 Old Gen: 7544250872 -> 8698295280; G1 Survivor Space:
> 2046820352 -
> INFO  11:46:29 G1 Young Generation GC in 355ms.  G1 Eden Space:
> 17230200832 -> 0; G1 Old Gen: 4769358328 -> 5362416624; G1 Survivor Space:
> 1761607680 -
> INFO  11:49:18 G1 Young Generation GC in 233ms.  G1 Eden Space:
> 17867735040 -> 0; G1 Old Gen: 6752829432 -> 6944805360; G1 Survivor Space:
> 1509949440 -
> INFO  11:52:24 G1 Young Generation GC in 209ms.  G1 Eden Space:
> 19193135104 -> 0; G1 Survivor Space: 1409286144 -> 1946157056;
> INFO  11:52:47 G1 Young Generation GC in 338ms.  G1 Eden Space:
> 18287165440 -> 0; G1 Old Gen: 8740881904 -> 9340426216; G1 Survivor Space:
> 1946157056 -
> INFO  12:04:08 G1 Young Generation GC in 810ms.  G1 Eden Space:
> 18119393280 -> 0; G1 Old Gen: 3471267832 -> 5003518440; G1 Survivor Space:
> 1325400064 -
> INFO  12:06:28 G1 Young Generation GC in 345ms.  G1 Eden Space: 8673820672
> -> 0; G1 Old Gen: 5318377952 -> 6577762784; G1 Survivor Space: 1409286144 ->
> INFO  12:09:31 G1 Young Generation GC in 204ms.  G1 Eden Space:
> 19159580672 -> 0; G1 Old Gen: 6577762784 -> 6586151384; G1 Survivor Space:
> 1442840576 -
> INFO  12:09:31 Redistributing index summaries
> INFO  12:10:01 G1 Young Generation GC in 298ms.  G1 Eden Space:
> 17230200832 -> 0; G1 Old Gen: 6586151384 -> 7710758368; G1 Survivor Space:
> 1946157056 -
> INFO  12:12:37 G1 Young Generation GC in 214ms.  G1 Eden Space:
> 18773704704 -> 0; G1 Old Gen: 8380219864 -> 8919123416; G1 Survivor Space:
> 1728053248 -
> INFO  12:24:54 G1 Young Generation GC in 299ms.  G1 Eden Space:
> 16055795712 -> 0; G1 Old Gen: 4375908856 -> 5009812976; G1 Survivor Space:
> 1392508928 -
> INFO  12:28:34 G1 Young Generation GC in 485ms.  G1 Eden Space:
> 13304332288 -> 0; G1 Old Gen: 6481607672 -> 7506585072; G1 Survivor Space:
> 1979711488 -
> INFO  12:44:52 G1 Young Generation GC in 261ms.  G1 Eden Space:
> 18773704704 -> 0; G1 Old Gen: 4239590896 -> 4692442608; G1 Survivor Space:
> 1828716544 -
> INFO  12:48:37 G1 Young Generation GC in 251ms.  G1 Eden Space:
> 13656653824 -> 0; G1 Old Gen: 6719275504 -> 7776254960; G1 Survivor Space:
> 1862270976 -
> INFO  12:53:47 G1 Young Generation GC in 217ms.  G1 Eden Space:
> 17548967936 -> 0; G1 Old Gen: 9667142640 -> 10202159608; G1 Survivor Space:
> 1845493760
> INFO  13:05:40 G1 Young Generation GC in 287ms.  G1 Eden Space:
> 18320719872 -> 0; G1 Old Gen: 5298861040 -> 5826686456; G1 Survivor Space:
> 1862270976 -
> INFO  13:09:31 Redistributing index summaries
> INFO  13:11:44 G1 Young Generation GC in 268ms.  G1 Eden Space:
> 10871635968 -> 0; G1 Old Gen: 8799649784 -> 9797878264; G1 Survivor Space:
> 1811939328 -
> INFO  13:14:22 G1 Young Generation GC in 202ms.  G1 Eden Space:
> 17750294528 -> 0; G1 Survivor Space: 1325400064 -> 1795162112;
> INFO  13:14:40 G1 Young Generation GC in 254ms.  G1 Eden Space:
> 14092861440 -> 0; G1 Old Gen: 10509551608 -> 11563311096; G1 Survivor
> Space: 1795162112
> INFO  13:16:33 G1 Young Generation GC in 213ms.  G1 Eden Space:
> 16039018496 -> 0; G1 Old Gen: 11619767792 -> 12115354104; G1 Survivor
> Space: 1627389952
> INFO  13:25:40 G1 Young Generation GC in 245ms.  G1 Eden Space:
> 18790481920 -> 0; G1 Old Gen: 4253024752 -> 4761733616; G1 Survivor Space:
> 1811939328 -
> INFO  13:27:38 G1 Young Generation GC in 208ms.  G1 Eden Space:
> 13958643712 -> 0; G1 Old Gen: 5377097720 -> 6427987960; G1 Survivor Space:
> 1912602624 -
> INFO  13:29:30 G1 Young Generation GC in 207ms.  G1 Eden Space:
> 12264144896 -> 0; G1 Old Gen: 6427987960 -> 7337681912; G1 Survivor Space:
> 1795162112 -
> INFO  13:33:32 G1 Young Generation GC in 202ms.  G1 Eden Space:
> 18589155328 -> 0; G1 Old Gen: 8506316280 -> 8514704880; G1 Survivor Space:
> 1308622848 -
> INFO  13:35:09 G1 Young Generation GC in 203ms.  G1 Eden Space:
> 17532190720 -> 0; G1 Survivor Space: 1258291200 -> 1761607680;
> INFO  13:38:33 G1 Young Generation GC in 216ms.  G1 Eden Space:
> 11106516992 -> 0; G1 Old Gen: 11571635704 -> 12698973168; G1 Survivor
> Space: 1795162112
> INFO  13:47:36 G1 Young Generation GC in 229ms.  G1 Eden Space:
> 19075694592 -> 0; G1 Old Gen: 3825205752 -> 4303660528; G1 Survivor Space:
> 1526726656 -
> INFO  13:53:43 G1 Young Generation GC in 207ms.  G1 Eden Space:
> 14042529792 -> 0; G1 Old Gen: 7324985848 -> 8363582960; G1 Survivor Space:
> 1660944384 -
> INFO  13:55:40 G1 Young Generation GC in 201ms.  G1 Eden Space:
> 18874368000 -> 0; G1 Old Gen: 8355194360 -> 8820888560; G1 Survivor Space:
> 1728053248 -
> INFO  13:56:22 G1 Young Generation GC in 206ms.  G1 Eden Space:
> 19394461696 -> 0; G1 Survivor Space: 1207959552 -> 1711276032;
> INFO  14:08:09 Saved KeyCache (4563 items) in 189 ms
> INFO  14:09:31 Redistributing index summaries
> INFO  14:12:47 G1 Young Generation GC in 271ms.  G1 Eden Space:
> 18874368000 -> 0; G1 Old Gen: 4345299448 -> 4814148600; G1 Survivor Space:
> 1728053248 -
> INFO  14:18:05 G1 Young Generation GC in 224ms.  G1 Eden Space:
> 18907922432 -> 0; G1 Old Gen: 7021265392 -> 7483297784; G1 Survivor Space:
> 1627389952 -
> INFO  14:36:05 G1 Young Generation GC in 304ms.  G1 Eden Space:
> 18958254080 -> 0; G1 Old Gen: 3976200696 -> 4437455344; G1 Survivor Space:
> 1644167168 -
> INFO  14:36:07 G1 Young Generation GC in 212ms.  G1 Eden Space: 1711276032
> -> 0; G1 Old Gen: 4437455344 -> 5830082552; G1 Survivor Space: 1459617792 ->
> INFO  15:00:30 G1 Young Generation GC in 233ms.  G1 Eden Space:
> 19159580672 -> 0; G1 Old Gen: 4040399864 -> 4343952368; G1 Survivor Space:
> 1442840576 -
> INFO  15:09:31 Redistributing index summaries
> INFO  15:26:28 G1 Young Generation GC in 234ms.  G1 Eden Space:
> 19075694592 -> 0; G1 Old Gen: 4604513264 -> 5064485872; G1 Survivor Space:
> 1526726656 -
> INFO  15:52:08 G1 Young Generation GC in 222ms.  G1 Eden Space:
> 19126026240 -> 0; G1 Old Gen: 4118806520 -> 4391313912; G1 Survivor Space:
> 1476395008 -
> INFO  15:55:06 G1 Young Generation GC in 202ms.  G1 Eden Space:
> 19142803456 -> 0; G1 Old Gen: 5267292152 -> 5559551472;
> INFO  16:09:31 Redistributing index summaries
> INFO  16:20:48 G1 Young Generation GC in 241ms.  G1 Eden Space:
> 19126026240 -> 0; G1 Old Gen: 3854587896 -> 4287600112; G1 Survivor Space:
> 1476395008 -
> INFO  16:31:07 G1 Young Generation GC in 403ms.  G1 Eden Space: 822083584
> -> 0; G1 Old Gen: 9219080688 -> 9994162168; G1 Survivor Space: 889192448 ->
> 2
> INFO  16:33:41 G1 Young Generation GC in 426ms.  G1 Eden Space: 620756992
> -> 0; G1 Old Gen: 10147301880 -> 11065889792; G1 Survivor Space: 1090519040
> -
> INFO  16:38:17 G1 Young Generation GC in 835ms.  G1 Eden Space: 251658240
> -> 0; G1 Old Gen: 12062818288 -> 13404581312; G1 Survivor Space: 1459617792
> -
> INFO  16:43:08 G1 Young Generation GC in 569ms.  G1 Eden Space: 3154116608
> -> 0; G1 Old Gen: 14361297352 -> 15461740984; G1 Survivor Space: 1258291200
> INFO  16:46:08 G1 Young Generation GC in 432ms.  G1 Eden Space:
> 18706595840 -> 0; G1 Old Gen: 3256274416 -> 4355653104; G1 Survivor Space:
> 1023410176 -
> INFO  17:09:31 Redistributing index summaries
> INFO  17:26:10 G1 Young Generation GC in 744ms.  G1 Eden Space: 201326592
> -> 0; G1 Old Gen: 6969713640 -> 8462801376; G1 Survivor Space: 1577058304
> ->
> INFO  17:29:04 G1 Young Generation GC in 204ms.  G1 Eden Space: 603979776
> -> 0; G1 Old Gen: 8454412776 -> 9439609320; G1 Survivor Space: 1107296256
> ->
> INFO  17:41:09 G1 Young Generation GC in 279ms.  G1 Eden Space: 352321536
> -> 0; G1 Old Gen: 13615267816 -> 14860613592; G1 Survivor Space: 1358954496
> -
> INFO  17:46:51 G1 Young Generation GC in 493ms.  G1 Eden Space: 754974720
> -> 0; G1 Old Gen: 3560839664 -> 4350490576; G1 Survivor Space: 956301312 ->
> 2
> INFO  17:49:09 G1 Young Generation GC in 290ms.  G1 Eden Space: 335544320
> -> 0; G1 Old Gen: 4350490576 -> 5603827664; G1 Survivor Space: 1375731712
> ->
> INFO  17:55:17 G1 Young Generation GC in 803ms.  G1 Eden Space: 201326592
> -> 0; G1 Old Gen: 6543114704 -> 8036691400; G1 Survivor Space: 1577058304
> ->
> INFO  17:58:07 G1 Young Generation GC in 206ms.  G1 Eden Space: 4764729344
> -> 0; G1 Old Gen: 8036691400 -> 8934491600; G1 Survivor Space: 973078528 ->
> INFO  18:07:29 G1 Young Generation GC in 306ms.  G1 Eden Space: 704643072
> -> 0; G1 Old Gen: 12163481536 -> 13045510592; G1 Survivor Space: 1006632960
> -
> INFO  18:08:09 Saved KeyCache (5307 items) in 85 ms
> INFO  18:09:31 Redistributing index summaries
> INFO  18:10:11 G1 Young Generation GC in 277ms.  G1 Eden Space: 402653184
> -> 0; G1 Old Gen: 13045510592 -> 14252493256; G1 Survivor Space: 1308622848
> -
> INFO  18:15:39 G1 Young Generation GC in 238ms.  G1 Eden Space:
> 13052674048 -> 0; G1 Old Gen: 15141437888 -> 15520873416; G1 Survivor
> Space: 1358954496
> INFO  18:20:16 G1 Young Generation GC in 540ms.  G1 Eden Space: 7616856064
> -> 0; G1 Old Gen: 3909694456 -> 4886048752; G1 Survivor Space: 1509949440 ->
> INFO  18:23:04 G1 Young Generation GC in 221ms.  G1 Eden Space:
> 16894656512 -> 0; G1 Old Gen: 5420089848 -> 6187322368; G1 Survivor Space:
> 939524096 ->
> INFO  18:42:08 G1 Young Generation GC in 708ms.  G1 Eden Space: 352321536
> -> 0; G1 Old Gen: 12840221688 -> 14085112816; G1 Survivor Space: 1358954496
> -
> INFO  18:50:13 G1 Young Generation GC in 318ms.  G1 Eden Space: 218103808
> -> 0; G1 Old Gen: 3631963120 -> 5033289720; G1 Survivor Space: 1493172224
> ->
> INFO  19:09:31 Redistributing index summaries
> INFO  19:27:02 G1 Young Generation GC in 384ms.  G1 Eden Space:
> 12901679104 -> 0; G1 Old Gen: 7034014192 -> 7759489528; G1 Survivor Space:
> 889192448 ->
> INFO  19:31:25 G1 Young Generation GC in 770ms.  G1 Eden Space: 218103808
> -> 0; G1 Old Gen: 8506048488 -> 9921951720; G1 Survivor Space: 1493172224
> ->
> INFO  20:09:31 Redistributing index summaries
> INFO  21:09:31 Redistributing index summaries
> INFO  21:19:39 G1 Young Generation GC in 201ms.  G1 Eden Space:
> 19142803456 -> 0; G1 Old Gen: 3917480432 -> 4124623344; G1 Survivor Space:
> 1459617792 -
> INFO  22:04:05 G1 Young Generation GC in 209ms.  G1 Eden Space:
> 19142803456 -> 0; G1 Old Gen: 4420796912 -> 4618141680; G1 Survivor Space:
> 1459617792 -
> INFO  22:07:49 G1 Young Generation GC in 215ms.  G1 Eden Space:
> 14713618432 -> 0; G1 Old Gen: 5970255352 -> 6555802608; G1 Survivor Space:
> 1493172224 -
> INFO  22:08:09 Saved KeyCache (5845 items) in 87 ms
> INFO  22:09:31 Redistributing index summaries
> INFO  22:38:04 G1 Young Generation GC in 201ms.  G1 Eden Space:
> 19075694592 -> 0; G1 Old Gen: 4248291312 -> 4459621368; G1 Survivor Space:
> 1526726656 -
> INFO  22:44:12 G1 Young Generation GC in 203ms.  G1 Eden Space:
> 10770972672 -> 0; G1 Old Gen: 6703617016 -> 7549747184; G1 Survivor Space:
> 1325400064 -
> INFO  22:47:05 G1 Young Generation GC in 207ms.  G1 Eden Space:
> 19025362944 -> 0; G1 Old Gen: 7549747184 -> 7771343352; G1 Survivor Space:
> 1577058304 -
> INFO  23:08:59 G1 Young Generation GC in 369ms.  G1 Eden Space:
> 15820914688 -> 0; G1 Old Gen: 4823449592 -> 5478502888; G1 Survivor Space:
> 1660944384 -
> INFO  23:09:31 Redistributing index summaries
> INFO  23:45:21 Maximum memory usage reached (2.000GiB), cannot allocate
> chunk of 1.000MiB
> INFO  23:58:40 G1 Young Generation GC in 283ms.  G1 Eden Space:
> 18941476864 -> 0; G1 Old Gen: 4831838192 -> 5297926640; G1 Survivor Space:
> 1660944384 -
> INFO  00:01:54 G1 Young Generation GC in 211ms.  G1 Eden Space:
> 18891145216 -> 0; G1 Old Gen: 6425673712 -> 6912948208; G1 Survivor Space:
> 1711276032 -
> INFO  00:05:37 Maximum memory usage reached (2.000GiB), cannot allocate
> chunk of 1.000MiB
> INFO  00:09:31 Redistributing index summaries
> INFO  00:14:33 G1 Young Generation GC in 282ms.  G1 Eden Space:
> 14344519680 -> 0; G1 Old Gen: 14159246832 -> 14677156848; G1 Survivor
> Space: 1207959552
> INFO  00:21:32 G1 Young Generation GC in 225ms.  G1 Eden Space:
> 19042140160 -> 0; G1 Old Gen: 4346926576 -> 4865982456; G1 Survivor Space:
> 1560281088 -
> INFO  00:22:09 Maximum memory usage reached (2.000GiB), cannot allocate
> chunk of 1.000MiB
> INFO  00:24:31 G1 Young Generation GC in 266ms.  G1 Eden Space:
> 19008585728 -> 0; G1 Old Gen: 6106907128 -> 6613907952; G1 Survivor Space:
> 1593835520 -
> INFO  00:29:46 G1 Young Generation GC in 233ms.  G1 Eden Space:
> 18236833792 -> 0; G1 Old Gen: 9076474360 -> 9498544624; G1 Survivor Space:
> 1795162112 -
> INFO  00:32:35 G1 Young Generation GC in 226ms.  G1 Eden Space:
> 16492003328 -> 0; G1 Old Gen: 10804527088 -> 11351712752; G1 Survivor
> Space: 1811939328
> INFO  00:37:09 Maximum memory usage reached (2.000GiB), cannot allocate
> chunk of 1.000MiB
> INFO  00:41:24 G1 Young Generation GC in 330ms.  G1 Eden Space:
> 19142803456 -> 0; G1 Old Gen: 3426117112 -> 3961507312; G1 Survivor Space:
> 1459617792 -
> INFO  00:44:26 G1 Young Generation GC in 325ms.  G1 Eden Space:
> 19042140160 -> 0; G1 Old Gen: 5268046320 -> 5788210120; G1 Survivor Space:
> 1560281088 -
> INFO  00:47:13 G1 Young Generation GC in 209ms.  G1 Eden Space:
> 19243466752 -> 0; G1 Old Gen: 7147093960 -> 7138705360; G1 Survivor Space:
> 1358954496 -
> INFO  00:47:28 G1 Young Generation GC in 312ms.  G1 Eden Space:
> 18605932544 -> 0; G1 Old Gen: 7138705360 -> 7710094280; G1 Survivor Space:
> 1996488704 -
> INFO  00:50:35 G1 Young Generation GC in 330ms.  G1 Eden Space:
> 17934843904 -> 0; G1 Old Gen: 9118416840 -> 9688820680; G1 Survivor Space:
> 1929379840 -
> INFO  00:52:52 Maximum memory usage reached (2.000GiB), cannot allocate
> chunk of 1.000MiB
> INFO  00:56:29 G1 Young Generation GC in 251ms.  G1 Eden Space:
> 14797504512 -> 0; G1 Old Gen: 13404995520 -> 13787049928; G1 Survivor
> Space: 1358954496
> INFO  00:58:14 G1 Young Generation GC in 662ms.  G1 Eden Space: 369098752
> -> 0; G1 Old Gen: 14730395584 -> 15984921520; G1 Survivor Space: 1342177280
> -
> INFO  01:01:25 G1 Young Generation GC in 288ms.  G1 Eden Space:
> 19042140160 -> 0; G1 Old Gen: 2618580968 -> 3127466464; G1 Survivor Space:
> 1560281088 -
> INFO  01:04:17 G1 Young Generation GC in 201ms.  G1 Eden Space:
> 19327352832 -> 0; G1 Old Gen: 4521459688 -> 4529848288; G1 Survivor Space:
> 1275068416 -
> INFO  01:04:39 G1 Young Generation GC in 283ms.  G1 Eden Space:
> 18723373056 -> 0; G1 Old Gen: 4529848288 -> 5024210408; G1 Survivor Space:
> 1879048192 -
> INFO  01:07:23 G1 Young Generation GC in 294ms.  G1 Eden Space:
> 18740150272 -> 0; G1 Old Gen: 6308233704 -> 6875745256; G1 Survivor Space:
> 1862270976 -
> INFO  01:07:53 Maximum memory usage reached (2.000GiB), cannot allocate
> chunk of 1.000MiB
> INFO  01:09:31 Redistributing index summaries
> INFO  01:10:12 G1 Young Generation GC in 213ms.  G1 Eden Space:
> 19243466752 -> 0; G1 Old Gen: 8262779360 -> 8254390760; G1 Survivor Space:
> 1358954496 -
> INFO  01:10:33 G1 Young Generation GC in 319ms.  G1 Eden Space:
> 18639486976 -> 0; G1 Old Gen: 8254390760 -> 8821123552; G1 Survivor Space:
> 1962934272 -
> INFO  01:22:55 Maximum memory usage reached (2.000GiB), cannot allocate
> chunk of 1.000MiB
> INFO  01:24:25 G1 Young Generation GC in 227ms.  G1 Eden Space:
> 19226689536 -> 0; G1 Old Gen: 4245307376 -> 4236918776; G1 Survivor Space:
> 1375731712 -
> INFO  01:24:48 G1 Young Generation GC in 219ms.  G1 Eden Space:
> 15401484288 -> 0; G1 Old Gen: 4236918776 -> 5231503864; G1 Survivor Space:
> 2097152000 -
> INFO  01:42:01 G1 Young Generation GC in 212ms.  G1 Eden Space:
> 16005464064 -> 0; G1 Old Gen: 4320133112 -> 5100782576; G1 Survivor Space:
> 1627389952 -
> INFO  01:48:28 G1 Young Generation GC in 209ms.  G1 Eden Space:
> 14730395648 -> 0; G1 Old Gen: 8901994480 -> 9402393592; G1 Survivor Space:
> 1291845632 -
> INFO  02:04:18 G1 Young Generation GC in 241ms.  G1 Eden Space:
> 19042140160 -> 0; G1 Old Gen: 7825270264 -> 8327173624; G1 Survivor Space:
> 1560281088 -
> INFO  02:08:09 Saved KeyCache (8634 items) in 94 ms
> INFO  02:09:31 Redistributing index summaries
> INFO  02:14:50 G1 Young Generation GC in 314ms.  G1 Eden Space:
> 19075694592 -> 0; G1 Old Gen: 3505807352 -> 4140375528; G1 Survivor Space:
> 1526726656 -
> INFO  02:48:35 G1 Young Generation GC in 321ms.  G1 Eden Space:
> 11576279040 -> 0; G1 Old Gen: 3865890800 -> 4482588664; G1 Survivor Space:
> 1543503872 -
> INFO  02:53:54 G1 Young Generation GC in 203ms.  G1 Eden Space:
> 17901289472 -> 0; G1 Old Gen: 7927235056 -> 8552357368; G1 Survivor Space:
> 1593835520 -
> INFO  03:04:33 G1 Young Generation GC in 345ms.  G1 Eden Space:
> 13505658880 -> 0; G1 Old Gen: 3765058032 -> 4443454952; G1 Survivor Space:
> 1442840576 -
> INFO  03:09:31 Redistributing index summaries
> INFO  03:22:56 G1 Young Generation GC in 250ms.  G1 Eden Space:
> 18991808512 -> 0; G1 Old Gen: 4806672880 -> 5340134904; G1 Survivor Space:
> 1610612736 -
> INFO  03:35:13 G1 Young Generation GC in 210ms.  G1 Eden Space:
> 14445182976 -> 0; G1 Survivor Space: 1073741824 -> 1744830464;
> INFO  03:38:42 G1 Young Generation GC in 214ms.  G1 Eden Space:
> 18303942656 -> 0; G1 Old Gen: 4786540536 -> 5432116720; G1 Survivor Space:
> 1375731712 -
> INFO  03:52:11 G1 Young Generation GC in 207ms.  G1 Eden Space: 117440512
> -> 0; G1 Old Gen: 12054831088 -> 10154180600; G1 Survivor Space: 1593835520
> -
> INFO  03:54:48 G1 Young Generation GC in 224ms.  G1 Eden Space:
> 16156459008 -> 0; G1 Old Gen: 4303355896 -> 4801558520; G1 Survivor Space:
> 1560281088 -
> INFO  04:09:31 Redistributing index summaries
> INFO  04:16:09 G1 Young Generation GC in 216ms.  G1 Eden Space:
> 19142803456 -> 0; G1 Old Gen: 5853311984 -> 6366188536; G1 Survivor Space:
> 1459617792 -
> INFO  04:19:13 G1 Young Generation GC in 229ms.  G1 Eden Space:
> 19042140160 -> 0; G1 Old Gen: 7826571256 -> 8457920504; G1 Survivor Space:
> 1560281088 -
> INFO  04:27:41 G1 Young Generation GC in 311ms.  G1 Eden Space:
> 13723762688 -> 0; G1 Old Gen: 14176747504 -> 14808230392; G1 Survivor
> Space: 1509949440
> INFO  04:29:56 G1 Young Generation GC in 274ms.  G1 Eden Space:
> 18740150272 -> 0; G1 Old Gen: 3376452080 -> 3818638328; G1 Survivor Space:
> 1459617792 -
> INFO  04:33:40 Maximum memory usage reached (2.000GiB), cannot allocate
> chunk of 1.000MiB
> INFO  04:49:07 G1 Young Generation GC in 319ms.  G1 Eden Space:
> 16005464064 -> 0; G1 Old Gen: 4435714040 -> 5023449072; G1 Survivor Space:
> 1442840576 -
> INFO  04:54:06 G1 Young Generation GC in 208ms.  G1 Eden Space:
> 18706595840 -> 0; G1 Old Gen: 7843348472 -> 8466889208; G1 Survivor Space:
> 1476395008 -
> INFO  05:09:31 Redistributing index summaries
>
>
> I found   INFO   Maximum memory usage reached (2.000GiB), cannot allocate
> chunk of 1.000MiB, does it  mean the  memory is not enough ?
>
>
> I want to know why  adding  a new node  spent  so  much time and what need
> I do to  complete  a new node  joining the cluster ?
>
>
> I need some help and really need some suggestions.
>
>
>
> Thanks  a lot!
>
>
>
>
>
>
> ------------------------------
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>
>
>

Mime
View raw message