cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "wxn002@zjqunshuo.com" <wxn...@zjqunshuo.com>
Subject Re: Cassandra cluster add new node slowly
Date Thu, 04 Jan 2018 02:29:30 GMT
Adding new node is really slow when you have a large load(for me, slow means several hours).
So I'm interested, is there anyway to speed up the addition when adding a new node?

Best Regards,
-Simon
 
发件人: qf zhou
发送时间: 2018-01-03 11:30
收件人: user@cassandra.apache.org
主题: Cassandra cluster add new node slowly
The cluster has  3 nodes,  and  the data in each node is  about 1.2 T.  I want to add two
new nodes to expand the cluster.

Following the instructions from the datastax  website, ie,  (http://docs.datastax.com/en/archived/cassandra/3.x/cassandra/operations/opsAddNodeToCluster.html),
 

I try to add one  node  to  the cluster.  However,  it  is  too slow  and time cost too  much.
 After about  24 hours,  it still didn’t  success.

I run the command: nodetool netstats  on the new node,  it  shows that:

(tb1fullwithstate2  is a big table and 90% of  the cluster data  is  in it.   Here I use CompactionStrategy:
TimeWindowCompactionStrategy).

/*.*.*.3
        Receiving 136 files, 328573794609 bytes total. Already received 8 files, 5774621188
bytes total
            tb1/tb1fullwithneweststatetest 3758271/3758271 bytes(100%) received from idx:0/*.*.*.3
            system_distributed/repair_history 57534/57534 bytes(100%) received from idx:0/*.*.*.3
            system_distributed/parent_repair_history 507660/507660 bytes(100%) received from
idx:0/*.*.*.3
            tb1/tb1_device_last_state_eachday 15754096/15754096 bytes(100%) received from
idx:0/*.*.*.3
            mytest1/tb1_test1 8143775/8143775 bytes(100%) received from idx:0/*.*.*.3
            tb1/tb1fullwithstate 2251191007/2251191007 bytes(100%) received from idx:0/*.*.*.3
            applocationinfo/weiyirong_app 2760/2760 bytes(100%) received from idx:0/*.*.*.3
            tb1/tb1fullwithstate2 3490748006/4909554503 bytes(71%) received from idx:0/*.*.*.3
            tb1/tb1fullwithneweststate 4458079/4458079 bytes(100%) received from idx:0/*.*.*.3
    /*.*.*.2
        Receiving 136 files, 336762487360 bytes total. Already received 3 files, 5695770181
bytes total
            system_distributed/repair_history 31684/31684 bytes(100%) received from idx:0/*.*.*.2
            tb1/tb1fullwithstate 908260516/908260516 bytes(100%) received from idx:0/*.*.*.2
            tb1/tb1fullwithstate2 4783622958/4990450588 bytes(95%) received from idx:0/*.*.*.2
            tb1/tb1fullwithneweststate 3855023/3855023 bytes(100%) received from idx:0/*.*.*.2
    /*.*.*.4
        Receiving 132 files, 236250553620 bytes total. Already received 10 files, 3117465128
bytes total
            mytest1/wordstest2 46/46 bytes(100%) received from idx:0/*.*.*.4
            tb1/tb1fullwithneweststatetest 3416891/3416891 bytes(100%) received from idx:0/*.*.*.4
            system_distributed/repair_history 39720/39720 bytes(100%) received from idx:0/*.*.*.4
            system_distributed/parent_repair_history 452250/452250 bytes(100%) received from
idx:0/*.*.*.4
            mytest1/weblogs 104/104 bytes(100%) received from idx:0/*.*.*.4
            tb1/tb1_device_last_state_eachday 12670998/12670998 bytes(100%) received from
idx:0/*.*.*.4
            mytest1/tb1_test1 3257952/3257952 bytes(100%) received from idx:0/*.*.*.4
            tb1/tb1fullwithstate 647702056/647702056 bytes(100%) received from idx:0/*.*.*.4
            applocationinfo/weiyirong_app 3509/3509 bytes(100%) received from idx:0/*.*.*.4
            tb1/tb1fullwithstate2 2446436305/3566562762 bytes(68%) received from idx:0/*.*.*.4
            tb1/tb1fullwithneweststate 3485297/3485297 bytes(100%) received from idx:0/*.*.*.4

 

check  the log in the  logs/system.log,  it shows that:


INFO  06:09:33 Updating topology for /*.*.*.2
INFO  06:09:33 Updating topology for /*.*.*.2
INFO  06:09:33 Node /*.*.*.3 has restarted, now UP
INFO  06:09:33 InetAddress /*.*.*.3 is now UP
INFO  06:09:33 Node /*.*.*.3 state jump to NORMAL
INFO  06:09:33 Updating topology for /*.*.*.3
INFO  06:09:33 Updating topology for /*.*.*.3
INFO  06:09:33 Handshaking version with /*.*.*.2
INFO  06:09:33 Handshaking version with /*.*.*.3
INFO  06:09:36 Handshaking version with /*.*.*.4
INFO  06:10:03 JOINING: Starting to bootstrap...
INFO  06:10:04 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Executing streaming plan for
Bootstrap
INFO  06:10:04 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Starting streaming to /*.*.*.4
INFO  06:10:04 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75, ID#0] Beginning stream session
with /*.*.*.4
INFO  06:10:04 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527790874-1.hints
INFO  06:10:04 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527790874-1.hints
to endpoint c5271e74-19a1-4cee-98d7-dc169cf87
INFO  06:10:11 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527800873-1.hints
INFO  06:10:11 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527800873-1.hints
to endpoint c5271e74-19a1-4cee-98d7-dc169cf87
INFO  06:10:12 Deleted hint file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527790881-1.hints
INFO  06:10:12 Finished hinted handoff of file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527790881-1.hints
to endpoint c555e44c-9590-4f45-aea4-f5eca6818
INFO  06:10:20 Deleted hint file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527800880-1.hints
INFO  06:10:20 Finished hinted handoff of file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527800880-1.hints
to endpoint c555e44c-9590-4f45-aea4-f5eca6818
INFO  06:10:21 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527790884-1.hints
INFO  06:10:21 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527790884-1.hints
to endpoint c623bbc0-9839-4d2d-8ff3-db7115719
INFO  06:10:28 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527800887-1.hints
INFO  06:10:28 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527800887-1.hints
to endpoint c623bbc0-9839-4d2d-8ff3-db7115719
INFO  06:10:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75 ID#0] Prepare completed. Receiving
132 files(220.025GiB), sending 0 files(0.000KiB)
INFO  06:10:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Starting streaming to /*.*.*.2
INFO  06:10:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75, ID#0] Beginning stream session
with /*.*.*.2
INFO  06:10:34 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527810873-1.hints
INFO  06:10:34 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527810873-1.hints
to endpoint c5271e74-19a1-4cee-98d7-dc169cf87
INFO  06:10:42 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527820873-1.hints
INFO  06:10:42 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527820873-1.hints
to endpoint c5271e74-19a1-4cee-98d7-dc169cf87
INFO  06:10:58 Deleted hint file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527810881-1.hints
INFO  06:10:58 Finished hinted handoff of file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527810881-1.hints
to endpoint c555e44c-9590-4f45-aea4-f5eca6818
INFO  06:11:03 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75 ID#0] Prepare completed. Receiving
136 files(313.635GiB), sending 0 files(0.000KiB)
INFO  06:11:03 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Starting streaming to /*.*.*.3
INFO  06:11:03 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75, ID#0] Beginning stream session
with /*.*.*.3
INFO  06:11:06 Deleted hint file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527830880-1.hints
INFO  06:11:06 Finished hinted handoff of file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527830880-1.hints
to endpoint c555e44c-9590-4f45-aea4-f5eca6818
INFO  06:11:29 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527810889-1.hints
INFO  06:11:29 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527810889-1.hints
to endpoint c623bbc0-9839-4d2d-8ff3-db7115719
INFO  06:11:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75 ID#0] Prepare completed. Receiving
136 files(306.008GiB), sending 0 files(0.000KiB)
INFO  06:11:36 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527840887-1.hints
INFO  06:11:36 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527840887-1.hints
to endpoint c623bbc0-9839-4d2d-8ff3-db7115719
INFO  06:11:49 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527830873-1.hints
INFO  06:11:49 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527830873-1.hints
to endpoint c5271e74-19a1-4cee-98d7-dc169cf87
INFO  06:11:56 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527850874-1.hints
INFO  06:11:56 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527850874-1.hints
to endpoint c5271e74-19a1-4cee-98d7-dc169cf87
INFO  06:12:26 Deleted hint file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527840880-1.hints
INFO  06:12:26 Finished hinted handoff of file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527840880-1.hints
to endpoint c555e44c-9590-4f45-aea4-f5eca6818
INFO  06:12:45 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527850883-1.hints
INFO  06:12:45 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527850883-1.hints
to endpoint c623bbc0-9839-4d2d-8ff3-db7115719
INFO  06:12:56 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527860874-1.hints
INFO  06:12:56 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527860874-1.hints
to endpoint c5271e74-19a1-4cee-98d7-dc169cf87
INFO  06:24:03 G1 Young Generation GC in 242ms.  G1 Eden Space: 18924699648 -> 0; G1 Old
Gen: 4360876536 -> 5095617016; G1 Survivor Space: 1677721600 -
INFO  06:26:06 G1 Young Generation GC in 215ms.  G1 Eden Space: 18220056576 -> 0; G1 Old
Gen: 5981077496 -> 6597610488; G1 Survivor Space: 1493172224 -
INFO  06:29:04 G1 Young Generation GC in 225ms.  G1 Eden Space: 18538823680 -> 0; G1 Old
Gen: 7910457840 -> 8860639728; G1 Survivor Space: 1895825408 -
INFO  06:33:22 G1 Young Generation GC in 214ms.  G1 Eden Space: 16559112192 -> 0; G1 Old
Gen: 11868321784 -> 11941600752; G1 Survivor Space: 1241513984
INFO  06:33:33 G1 Young Generation GC in 205ms.  G1 Eden Space: 8069840896 -> 0; G1 Old
Gen: 11941600752 -> 13683606520; G1 Survivor Space: 1895825408 
INFO  06:41:01 G1 Young Generation GC in 383ms.  G1 Eden Space: 18874368000 -> 0; G1 Old
Gen: 4770399736 -> 5551887344; G1 Survivor Space: 1728053248 -
INFO  06:45:07 G1 Young Generation GC in 226ms.  G1 Eden Space: 19025362944 -> 0; G1 Old
Gen: 8036286448 -> 8739698672; G1 Survivor Space: 1577058304 -
INFO  06:47:11 G1 Young Generation GC in 203ms.  G1 Eden Space: 18538823680 -> 0; G1 Survivor
Space: 1191182336 -> 1778384896; 
INFO  06:47:22 G1 Young Generation GC in 241ms.  G1 Eden Space: 8086618112 -> 0; G1 Old
Gen: 9999220720 -> 11591876088; G1 Survivor Space: 1778384896 -
INFO  06:49:43 G1 Young Generation GC in 201ms.  G1 Eden Space: 16022241280 -> 0; G1 Old
Gen: 11591876088 -> 12084766712; G1 Survivor Space: 1644167168
INFO  06:53:21 G1 Young Generation GC in 322ms.  G1 Eden Space: 13237223424 -> 0; G1 Old
Gen: 14755561464 -> 15445886448; G1 Survivor Space: 1459617792
INFO  06:55:34 G1 Young Generation GC in 343ms.  G1 Eden Space: 16525557760 -> 0; G1 Old
Gen: 2939907056 -> 3636770792; G1 Survivor Space: 1610612736 -
INFO  07:09:30 Redistributing index summaries
INFO  07:14:25 G1 Young Generation GC in 270ms.  G1 Eden Space: 15602810880 -> 0; G1 Old
Gen: 4360518136 -> 5243274224; G1 Survivor Space: 1610612736 -
INFO  07:16:51 G1 Young Generation GC in 229ms.  G1 Eden Space: 19008585728 -> 0; G1 Old
Gen: 6132072440 -> 6815145976; G1 Survivor Space: 1593835520 -
INFO  07:19:34 G1 Young Generation GC in 215ms.  G1 Eden Space: 18991808512 -> 0; G1 Old
Gen: 8689337336 -> 9071161328; G1 Survivor Space: 1610612736 -
INFO  07:23:35 G1 Young Generation GC in 293ms.  G1 Eden Space: 15753805824 -> 0; G1 Old
Gen: 11777605616 -> 12289649640; G1 Survivor Space: 1694498816
INFO  07:29:42 G1 Young Generation GC in 362ms.  G1 Eden Space: 15955132416 -> 0; G1 Old
Gen: 3742637560 -> 4410361824; G1 Survivor Space: 1644167168 -
INFO  07:35:12 G1 Young Generation GC in 208ms.  G1 Eden Space: 19478347776 -> 0; G1 Old
Gen: 7199737832 -> 7208126432; G1 Survivor Space: 1124073472 -
INFO  07:35:33 G1 Young Generation GC in 219ms.  G1 Eden Space: 18740150272 -> 0; G1 Old
Gen: 7208126432 -> 8072478184; G1 Survivor Space: 1862270976 -
INFO  07:45:19 G1 Young Generation GC in 213ms.  G1 Eden Space: 13153337344 -> 0; G1 Old
Gen: 15418261472 -> 16123805152; G1 Survivor Space: 1174405120
INFO  07:48:08 G1 Young Generation GC in 335ms.  G1 Eden Space: 18387828736 -> 0; G1 Old
Gen: 3852453368 -> 4464797168; G1 Survivor Space: 1644167168 -
INFO  07:48:08 G1 Young Generation GC in 209ms.  G1 Eden Space: 352321536 -> 0; G1 Old
Gen: 4464797168 -> 5989466096; G1 Survivor Space: 1476395008 -> 
INFO  07:50:25 G1 Young Generation GC in 221ms.  G1 Eden Space: 17498636288 -> 0; G1 Old
Gen: 5981077496 -> 6488668656; G1 Survivor Space: 1593835520 -
INFO  08:03:31 G1 Young Generation GC in 748ms.  G1 Eden Space: 8673820672 -> 0; G1 Old
Gen: 4386367472 -> 5874747352; G1 Survivor Space: 1627389952 ->
INFO  08:09:30 Redistributing index summaries
INFO  08:10:05 G1 Young Generation GC in 213ms.  G1 Eden Space: 18220056576 -> 0; G1 Old
Gen: 9276024272 -> 9969348048; G1 Survivor Space: 1711276032 -
INFO  08:18:59 G1 Young Generation GC in 228ms.  G1 Eden Space: 17347641344 -> 0; G1 Old
Gen: 4546626040 -> 5180788216; G1 Survivor Space: 1593835520 -
INFO  08:33:12 G1 Young Generation GC in 207ms.  G1 Eden Space: 19411238912 -> 0; G1 Survivor
Space: 1191182336 -> 1828716544; 
INFO  08:33:13 G1 Young Generation GC in 273ms.  G1 Eden Space: 452984832 -> 0; G1 Old
Gen: 4668340728 -> 6320276984; G1 Survivor Space: 1828716544 -> 
INFO  08:41:59 G1 Young Generation GC in 202ms.  G1 Eden Space: 19025362944 -> 0; G1 Old
Gen: 8701765104 -> 8927144944; G1 Survivor Space: 1577058304 -
INFO  08:56:31 G1 Young Generation GC in 205ms.  G1 Eden Space: 11072962560 -> 0; G1 Old
Gen: 9563013104 -> 11068430328; G1 Survivor Space: 1660944384 
INFO  09:05:49 G1 Young Generation GC in 367ms.  G1 Eden Space: 18522046464 -> 0; G1 Old
Gen: 4009754608 -> 4732347872; G1 Survivor Space: 1778384896 -
INFO  09:09:30 Redistributing index summaries
INFO  09:14:52 G1 Young Generation GC in 213ms.  G1 Eden Space: 15267266560 -> 0; G1 Old
Gen: 12155092968 -> 12906235360; G1 Survivor Space: 1744830464
INFO  09:17:23 G1 Young Generation GC in 203ms.  G1 Eden Space: 14159970304 -> 0; G1 Old
Gen: 14779568608 -> 14771180008; G1 Survivor Space: 989855744 
INFO  09:17:45 G1 Young Generation GC in 204ms.  G1 Eden Space: 12616466432 -> 0; G1 Old
Gen: 14771180008 -> 15500684768; G1 Survivor Space: 1761607680
INFO  09:20:52 G1 Young Generation GC in 361ms.  G1 Eden Space: 19025362944 -> 0; G1 Old
Gen: 4351990768 -> 5086824424; G1 Survivor Space: 1577058304 -
INFO  09:23:51 G1 Young Generation GC in 207ms.  G1 Eden Space: 18891145216 -> 0; G1 Old
Gen: 7001879024 -> 7716560880; G1 Survivor Space: 1711276032 -
INFO  09:29:20 G1 Young Generation GC in 202ms.  G1 Eden Space: 17028874240 -> 0; G1 Survivor
Space: 1040187392 -> 1828716544; 
INFO  09:29:40 G1 Young Generation GC in 219ms.  G1 Eden Space: 15451815936 -> 0; G1 Old
Gen: 11802771944 -> 12574988776; G1 Survivor Space: 1828716544
INFO  09:44:51 G1 Young Generation GC in 202ms.  G1 Eden Space: 18001952768 -> 0; G1 Old
Gen: 9325481456 -> 10085596664; G1 Survivor Space: 1795162112 
INFO  09:45:23 G1 Young Generation GC in 209ms.  G1 Eden Space: 9177137152 -> 0; G1 Old
Gen: 10093985264 -> 12309118960; G1 Survivor Space: 1677721600 
INFO  09:50:33 G1 Young Generation GC in 201ms.  G1 Eden Space: 12901679104 -> 0; G1 Old
Gen: 14487498224 -> 15268560880; G1 Survivor Space: 1761607680
INFO  10:08:09 Saved KeyCache (2989 items) in 92 ms
INFO  10:09:30 Redistributing index summaries
INFO  10:22:17 G1 Young Generation GC in 216ms.  G1 Eden Space: 19495124992 -> 0; G1 Old
Gen: 7742571504 -> 7734182904; G1 Survivor Space: 1107296256 -
INFO  10:25:31 G1 Young Generation GC in 206ms.  G1 Eden Space: 18622709760 -> 0; G1 Survivor
Space: 1040187392 -> 1711276032; 
INFO  10:34:54 G1 Young Generation GC in 251ms.  G1 Eden Space: 18991808512 -> 0; G1 Old
Gen: 5066719736 -> 5691775992; G1 Survivor Space: 1610612736 -
INFO  10:40:08 G1 Young Generation GC in 209ms.  G1 Eden Space: 19025362944 -> 0; G1 Old
Gen: 9000976880 -> 9090653168; G1 Survivor Space: 1442840576 -
INFO  10:40:08 G1 Young Generation GC in 223ms.  G1 Eden Space: 469762048 -> 0; G1 Old
Gen: 9090653168 -> 11140124152; G1 Survivor Space: 1862270976 ->
INFO  10:52:15 G1 Young Generation GC in 219ms.  G1 Eden Space: 15820914688 -> 0; G1 Old
Gen: 5331537912 -> 5828939768; G1 Survivor Space: 1275068416 -
INFO  10:55:30 G1 Young Generation GC in 207ms.  G1 Eden Space: 15804137472 -> 0; G1 Old
Gen: 7501364216 -> 8001282032; G1 Survivor Space: 1459617792 -
INFO  11:07:57 G1 Young Generation GC in 295ms.  G1 Eden Space: 19159580672 -> 0; G1 Old
Gen: 4301379064 -> 4919709168; G1 Survivor Space: 1442840576 -
INFO  11:09:30 Redistributing index summaries
INFO  11:13:06 G1 Young Generation GC in 204ms.  G1 Eden Space: 19126026240 -> 0; G1 Old
Gen: 8044675064 -> 8659737080; G1 Survivor Space: 1476395008 -
INFO  11:27:07 G1 Young Generation GC in 202ms.  G1 Eden Space: 19159580672 -> 0; G1 Old
Gen: 4328521712 -> 4905718776; G1 Survivor Space: 1442840576 -
INFO  11:30:08 G1 Young Generation GC in 209ms.  G1 Eden Space: 19243466752 -> 0; G1 Survivor
Space: 1358954496 -> 1828716544; 
INFO  11:30:25 G1 Young Generation GC in 226ms.  G1 Eden Space: 14277410816 -> 0; G1 Old
Gen: 6349296624 -> 7552639472; G1 Survivor Space: 1828716544 -
INFO  11:32:30 G1 Young Generation GC in 215ms.  G1 Eden Space: 19142803456 -> 0; G1 Survivor
Space: 1459617792 -> 2046820352; 
INFO  11:33:01 G1 Young Generation GC in 247ms.  G1 Eden Space: 17263755264 -> 0; G1 Old
Gen: 7544250872 -> 8698295280; G1 Survivor Space: 2046820352 -
INFO  11:46:29 G1 Young Generation GC in 355ms.  G1 Eden Space: 17230200832 -> 0; G1 Old
Gen: 4769358328 -> 5362416624; G1 Survivor Space: 1761607680 -
INFO  11:49:18 G1 Young Generation GC in 233ms.  G1 Eden Space: 17867735040 -> 0; G1 Old
Gen: 6752829432 -> 6944805360; G1 Survivor Space: 1509949440 -
INFO  11:52:24 G1 Young Generation GC in 209ms.  G1 Eden Space: 19193135104 -> 0; G1 Survivor
Space: 1409286144 -> 1946157056; 
INFO  11:52:47 G1 Young Generation GC in 338ms.  G1 Eden Space: 18287165440 -> 0; G1 Old
Gen: 8740881904 -> 9340426216; G1 Survivor Space: 1946157056 -
INFO  12:04:08 G1 Young Generation GC in 810ms.  G1 Eden Space: 18119393280 -> 0; G1 Old
Gen: 3471267832 -> 5003518440; G1 Survivor Space: 1325400064 -
INFO  12:06:28 G1 Young Generation GC in 345ms.  G1 Eden Space: 8673820672 -> 0; G1 Old
Gen: 5318377952 -> 6577762784; G1 Survivor Space: 1409286144 ->
INFO  12:09:31 G1 Young Generation GC in 204ms.  G1 Eden Space: 19159580672 -> 0; G1 Old
Gen: 6577762784 -> 6586151384; G1 Survivor Space: 1442840576 -
INFO  12:09:31 Redistributing index summaries
INFO  12:10:01 G1 Young Generation GC in 298ms.  G1 Eden Space: 17230200832 -> 0; G1 Old
Gen: 6586151384 -> 7710758368; G1 Survivor Space: 1946157056 -
INFO  12:12:37 G1 Young Generation GC in 214ms.  G1 Eden Space: 18773704704 -> 0; G1 Old
Gen: 8380219864 -> 8919123416; G1 Survivor Space: 1728053248 -
INFO  12:24:54 G1 Young Generation GC in 299ms.  G1 Eden Space: 16055795712 -> 0; G1 Old
Gen: 4375908856 -> 5009812976; G1 Survivor Space: 1392508928 -
INFO  12:28:34 G1 Young Generation GC in 485ms.  G1 Eden Space: 13304332288 -> 0; G1 Old
Gen: 6481607672 -> 7506585072; G1 Survivor Space: 1979711488 -
INFO  12:44:52 G1 Young Generation GC in 261ms.  G1 Eden Space: 18773704704 -> 0; G1 Old
Gen: 4239590896 -> 4692442608; G1 Survivor Space: 1828716544 -
INFO  12:48:37 G1 Young Generation GC in 251ms.  G1 Eden Space: 13656653824 -> 0; G1 Old
Gen: 6719275504 -> 7776254960; G1 Survivor Space: 1862270976 -
INFO  12:53:47 G1 Young Generation GC in 217ms.  G1 Eden Space: 17548967936 -> 0; G1 Old
Gen: 9667142640 -> 10202159608; G1 Survivor Space: 1845493760 
INFO  13:05:40 G1 Young Generation GC in 287ms.  G1 Eden Space: 18320719872 -> 0; G1 Old
Gen: 5298861040 -> 5826686456; G1 Survivor Space: 1862270976 -
INFO  13:09:31 Redistributing index summaries
INFO  13:11:44 G1 Young Generation GC in 268ms.  G1 Eden Space: 10871635968 -> 0; G1 Old
Gen: 8799649784 -> 9797878264; G1 Survivor Space: 1811939328 -
INFO  13:14:22 G1 Young Generation GC in 202ms.  G1 Eden Space: 17750294528 -> 0; G1 Survivor
Space: 1325400064 -> 1795162112; 
INFO  13:14:40 G1 Young Generation GC in 254ms.  G1 Eden Space: 14092861440 -> 0; G1 Old
Gen: 10509551608 -> 11563311096; G1 Survivor Space: 1795162112
INFO  13:16:33 G1 Young Generation GC in 213ms.  G1 Eden Space: 16039018496 -> 0; G1 Old
Gen: 11619767792 -> 12115354104; G1 Survivor Space: 1627389952
INFO  13:25:40 G1 Young Generation GC in 245ms.  G1 Eden Space: 18790481920 -> 0; G1 Old
Gen: 4253024752 -> 4761733616; G1 Survivor Space: 1811939328 -
INFO  13:27:38 G1 Young Generation GC in 208ms.  G1 Eden Space: 13958643712 -> 0; G1 Old
Gen: 5377097720 -> 6427987960; G1 Survivor Space: 1912602624 -
INFO  13:29:30 G1 Young Generation GC in 207ms.  G1 Eden Space: 12264144896 -> 0; G1 Old
Gen: 6427987960 -> 7337681912; G1 Survivor Space: 1795162112 -
INFO  13:33:32 G1 Young Generation GC in 202ms.  G1 Eden Space: 18589155328 -> 0; G1 Old
Gen: 8506316280 -> 8514704880; G1 Survivor Space: 1308622848 -
INFO  13:35:09 G1 Young Generation GC in 203ms.  G1 Eden Space: 17532190720 -> 0; G1 Survivor
Space: 1258291200 -> 1761607680; 
INFO  13:38:33 G1 Young Generation GC in 216ms.  G1 Eden Space: 11106516992 -> 0; G1 Old
Gen: 11571635704 -> 12698973168; G1 Survivor Space: 1795162112
INFO  13:47:36 G1 Young Generation GC in 229ms.  G1 Eden Space: 19075694592 -> 0; G1 Old
Gen: 3825205752 -> 4303660528; G1 Survivor Space: 1526726656 -
INFO  13:53:43 G1 Young Generation GC in 207ms.  G1 Eden Space: 14042529792 -> 0; G1 Old
Gen: 7324985848 -> 8363582960; G1 Survivor Space: 1660944384 -
INFO  13:55:40 G1 Young Generation GC in 201ms.  G1 Eden Space: 18874368000 -> 0; G1 Old
Gen: 8355194360 -> 8820888560; G1 Survivor Space: 1728053248 -
INFO  13:56:22 G1 Young Generation GC in 206ms.  G1 Eden Space: 19394461696 -> 0; G1 Survivor
Space: 1207959552 -> 1711276032; 
INFO  14:08:09 Saved KeyCache (4563 items) in 189 ms
INFO  14:09:31 Redistributing index summaries
INFO  14:12:47 G1 Young Generation GC in 271ms.  G1 Eden Space: 18874368000 -> 0; G1 Old
Gen: 4345299448 -> 4814148600; G1 Survivor Space: 1728053248 -
INFO  14:18:05 G1 Young Generation GC in 224ms.  G1 Eden Space: 18907922432 -> 0; G1 Old
Gen: 7021265392 -> 7483297784; G1 Survivor Space: 1627389952 -
INFO  14:36:05 G1 Young Generation GC in 304ms.  G1 Eden Space: 18958254080 -> 0; G1 Old
Gen: 3976200696 -> 4437455344; G1 Survivor Space: 1644167168 -
INFO  14:36:07 G1 Young Generation GC in 212ms.  G1 Eden Space: 1711276032 -> 0; G1 Old
Gen: 4437455344 -> 5830082552; G1 Survivor Space: 1459617792 ->
INFO  15:00:30 G1 Young Generation GC in 233ms.  G1 Eden Space: 19159580672 -> 0; G1 Old
Gen: 4040399864 -> 4343952368; G1 Survivor Space: 1442840576 -
INFO  15:09:31 Redistributing index summaries
INFO  15:26:28 G1 Young Generation GC in 234ms.  G1 Eden Space: 19075694592 -> 0; G1 Old
Gen: 4604513264 -> 5064485872; G1 Survivor Space: 1526726656 -
INFO  15:52:08 G1 Young Generation GC in 222ms.  G1 Eden Space: 19126026240 -> 0; G1 Old
Gen: 4118806520 -> 4391313912; G1 Survivor Space: 1476395008 -
INFO  15:55:06 G1 Young Generation GC in 202ms.  G1 Eden Space: 19142803456 -> 0; G1 Old
Gen: 5267292152 -> 5559551472; 
INFO  16:09:31 Redistributing index summaries
INFO  16:20:48 G1 Young Generation GC in 241ms.  G1 Eden Space: 19126026240 -> 0; G1 Old
Gen: 3854587896 -> 4287600112; G1 Survivor Space: 1476395008 -
INFO  16:31:07 G1 Young Generation GC in 403ms.  G1 Eden Space: 822083584 -> 0; G1 Old
Gen: 9219080688 -> 9994162168; G1 Survivor Space: 889192448 -> 2
INFO  16:33:41 G1 Young Generation GC in 426ms.  G1 Eden Space: 620756992 -> 0; G1 Old
Gen: 10147301880 -> 11065889792; G1 Survivor Space: 1090519040 -
INFO  16:38:17 G1 Young Generation GC in 835ms.  G1 Eden Space: 251658240 -> 0; G1 Old
Gen: 12062818288 -> 13404581312; G1 Survivor Space: 1459617792 -
INFO  16:43:08 G1 Young Generation GC in 569ms.  G1 Eden Space: 3154116608 -> 0; G1 Old
Gen: 14361297352 -> 15461740984; G1 Survivor Space: 1258291200 
INFO  16:46:08 G1 Young Generation GC in 432ms.  G1 Eden Space: 18706595840 -> 0; G1 Old
Gen: 3256274416 -> 4355653104; G1 Survivor Space: 1023410176 -
INFO  17:09:31 Redistributing index summaries
INFO  17:26:10 G1 Young Generation GC in 744ms.  G1 Eden Space: 201326592 -> 0; G1 Old
Gen: 6969713640 -> 8462801376; G1 Survivor Space: 1577058304 -> 
INFO  17:29:04 G1 Young Generation GC in 204ms.  G1 Eden Space: 603979776 -> 0; G1 Old
Gen: 8454412776 -> 9439609320; G1 Survivor Space: 1107296256 -> 
INFO  17:41:09 G1 Young Generation GC in 279ms.  G1 Eden Space: 352321536 -> 0; G1 Old
Gen: 13615267816 -> 14860613592; G1 Survivor Space: 1358954496 -
INFO  17:46:51 G1 Young Generation GC in 493ms.  G1 Eden Space: 754974720 -> 0; G1 Old
Gen: 3560839664 -> 4350490576; G1 Survivor Space: 956301312 -> 2
INFO  17:49:09 G1 Young Generation GC in 290ms.  G1 Eden Space: 335544320 -> 0; G1 Old
Gen: 4350490576 -> 5603827664; G1 Survivor Space: 1375731712 -> 
INFO  17:55:17 G1 Young Generation GC in 803ms.  G1 Eden Space: 201326592 -> 0; G1 Old
Gen: 6543114704 -> 8036691400; G1 Survivor Space: 1577058304 -> 
INFO  17:58:07 G1 Young Generation GC in 206ms.  G1 Eden Space: 4764729344 -> 0; G1 Old
Gen: 8036691400 -> 8934491600; G1 Survivor Space: 973078528 -> 
INFO  18:07:29 G1 Young Generation GC in 306ms.  G1 Eden Space: 704643072 -> 0; G1 Old
Gen: 12163481536 -> 13045510592; G1 Survivor Space: 1006632960 -
INFO  18:08:09 Saved KeyCache (5307 items) in 85 ms
INFO  18:09:31 Redistributing index summaries
INFO  18:10:11 G1 Young Generation GC in 277ms.  G1 Eden Space: 402653184 -> 0; G1 Old
Gen: 13045510592 -> 14252493256; G1 Survivor Space: 1308622848 -
INFO  18:15:39 G1 Young Generation GC in 238ms.  G1 Eden Space: 13052674048 -> 0; G1 Old
Gen: 15141437888 -> 15520873416; G1 Survivor Space: 1358954496
INFO  18:20:16 G1 Young Generation GC in 540ms.  G1 Eden Space: 7616856064 -> 0; G1 Old
Gen: 3909694456 -> 4886048752; G1 Survivor Space: 1509949440 ->
INFO  18:23:04 G1 Young Generation GC in 221ms.  G1 Eden Space: 16894656512 -> 0; G1 Old
Gen: 5420089848 -> 6187322368; G1 Survivor Space: 939524096 ->
INFO  18:42:08 G1 Young Generation GC in 708ms.  G1 Eden Space: 352321536 -> 0; G1 Old
Gen: 12840221688 -> 14085112816; G1 Survivor Space: 1358954496 -
INFO  18:50:13 G1 Young Generation GC in 318ms.  G1 Eden Space: 218103808 -> 0; G1 Old
Gen: 3631963120 -> 5033289720; G1 Survivor Space: 1493172224 -> 
INFO  19:09:31 Redistributing index summaries
INFO  19:27:02 G1 Young Generation GC in 384ms.  G1 Eden Space: 12901679104 -> 0; G1 Old
Gen: 7034014192 -> 7759489528; G1 Survivor Space: 889192448 ->
INFO  19:31:25 G1 Young Generation GC in 770ms.  G1 Eden Space: 218103808 -> 0; G1 Old
Gen: 8506048488 -> 9921951720; G1 Survivor Space: 1493172224 -> 
INFO  20:09:31 Redistributing index summaries
INFO  21:09:31 Redistributing index summaries
INFO  21:19:39 G1 Young Generation GC in 201ms.  G1 Eden Space: 19142803456 -> 0; G1 Old
Gen: 3917480432 -> 4124623344; G1 Survivor Space: 1459617792 -
INFO  22:04:05 G1 Young Generation GC in 209ms.  G1 Eden Space: 19142803456 -> 0; G1 Old
Gen: 4420796912 -> 4618141680; G1 Survivor Space: 1459617792 -
INFO  22:07:49 G1 Young Generation GC in 215ms.  G1 Eden Space: 14713618432 -> 0; G1 Old
Gen: 5970255352 -> 6555802608; G1 Survivor Space: 1493172224 -
INFO  22:08:09 Saved KeyCache (5845 items) in 87 ms
INFO  22:09:31 Redistributing index summaries
INFO  22:38:04 G1 Young Generation GC in 201ms.  G1 Eden Space: 19075694592 -> 0; G1 Old
Gen: 4248291312 -> 4459621368; G1 Survivor Space: 1526726656 -
INFO  22:44:12 G1 Young Generation GC in 203ms.  G1 Eden Space: 10770972672 -> 0; G1 Old
Gen: 6703617016 -> 7549747184; G1 Survivor Space: 1325400064 -
INFO  22:47:05 G1 Young Generation GC in 207ms.  G1 Eden Space: 19025362944 -> 0; G1 Old
Gen: 7549747184 -> 7771343352; G1 Survivor Space: 1577058304 -
INFO  23:08:59 G1 Young Generation GC in 369ms.  G1 Eden Space: 15820914688 -> 0; G1 Old
Gen: 4823449592 -> 5478502888; G1 Survivor Space: 1660944384 -
INFO  23:09:31 Redistributing index summaries
INFO  23:45:21 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB
INFO  23:58:40 G1 Young Generation GC in 283ms.  G1 Eden Space: 18941476864 -> 0; G1 Old
Gen: 4831838192 -> 5297926640; G1 Survivor Space: 1660944384 -
INFO  00:01:54 G1 Young Generation GC in 211ms.  G1 Eden Space: 18891145216 -> 0; G1 Old
Gen: 6425673712 -> 6912948208; G1 Survivor Space: 1711276032 -
INFO  00:05:37 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB
INFO  00:09:31 Redistributing index summaries
INFO  00:14:33 G1 Young Generation GC in 282ms.  G1 Eden Space: 14344519680 -> 0; G1 Old
Gen: 14159246832 -> 14677156848; G1 Survivor Space: 1207959552
INFO  00:21:32 G1 Young Generation GC in 225ms.  G1 Eden Space: 19042140160 -> 0; G1 Old
Gen: 4346926576 -> 4865982456; G1 Survivor Space: 1560281088 -
INFO  00:22:09 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB
INFO  00:24:31 G1 Young Generation GC in 266ms.  G1 Eden Space: 19008585728 -> 0; G1 Old
Gen: 6106907128 -> 6613907952; G1 Survivor Space: 1593835520 -
INFO  00:29:46 G1 Young Generation GC in 233ms.  G1 Eden Space: 18236833792 -> 0; G1 Old
Gen: 9076474360 -> 9498544624; G1 Survivor Space: 1795162112 -
INFO  00:32:35 G1 Young Generation GC in 226ms.  G1 Eden Space: 16492003328 -> 0; G1 Old
Gen: 10804527088 -> 11351712752; G1 Survivor Space: 1811939328
INFO  00:37:09 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB
INFO  00:41:24 G1 Young Generation GC in 330ms.  G1 Eden Space: 19142803456 -> 0; G1 Old
Gen: 3426117112 -> 3961507312; G1 Survivor Space: 1459617792 -
INFO  00:44:26 G1 Young Generation GC in 325ms.  G1 Eden Space: 19042140160 -> 0; G1 Old
Gen: 5268046320 -> 5788210120; G1 Survivor Space: 1560281088 -
INFO  00:47:13 G1 Young Generation GC in 209ms.  G1 Eden Space: 19243466752 -> 0; G1 Old
Gen: 7147093960 -> 7138705360; G1 Survivor Space: 1358954496 -
INFO  00:47:28 G1 Young Generation GC in 312ms.  G1 Eden Space: 18605932544 -> 0; G1 Old
Gen: 7138705360 -> 7710094280; G1 Survivor Space: 1996488704 -
INFO  00:50:35 G1 Young Generation GC in 330ms.  G1 Eden Space: 17934843904 -> 0; G1 Old
Gen: 9118416840 -> 9688820680; G1 Survivor Space: 1929379840 -
INFO  00:52:52 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB
INFO  00:56:29 G1 Young Generation GC in 251ms.  G1 Eden Space: 14797504512 -> 0; G1 Old
Gen: 13404995520 -> 13787049928; G1 Survivor Space: 1358954496
INFO  00:58:14 G1 Young Generation GC in 662ms.  G1 Eden Space: 369098752 -> 0; G1 Old
Gen: 14730395584 -> 15984921520; G1 Survivor Space: 1342177280 -
INFO  01:01:25 G1 Young Generation GC in 288ms.  G1 Eden Space: 19042140160 -> 0; G1 Old
Gen: 2618580968 -> 3127466464; G1 Survivor Space: 1560281088 -
INFO  01:04:17 G1 Young Generation GC in 201ms.  G1 Eden Space: 19327352832 -> 0; G1 Old
Gen: 4521459688 -> 4529848288; G1 Survivor Space: 1275068416 -
INFO  01:04:39 G1 Young Generation GC in 283ms.  G1 Eden Space: 18723373056 -> 0; G1 Old
Gen: 4529848288 -> 5024210408; G1 Survivor Space: 1879048192 -
INFO  01:07:23 G1 Young Generation GC in 294ms.  G1 Eden Space: 18740150272 -> 0; G1 Old
Gen: 6308233704 -> 6875745256; G1 Survivor Space: 1862270976 -
INFO  01:07:53 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB
INFO  01:09:31 Redistributing index summaries
INFO  01:10:12 G1 Young Generation GC in 213ms.  G1 Eden Space: 19243466752 -> 0; G1 Old
Gen: 8262779360 -> 8254390760; G1 Survivor Space: 1358954496 -
INFO  01:10:33 G1 Young Generation GC in 319ms.  G1 Eden Space: 18639486976 -> 0; G1 Old
Gen: 8254390760 -> 8821123552; G1 Survivor Space: 1962934272 -
INFO  01:22:55 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB
INFO  01:24:25 G1 Young Generation GC in 227ms.  G1 Eden Space: 19226689536 -> 0; G1 Old
Gen: 4245307376 -> 4236918776; G1 Survivor Space: 1375731712 -
INFO  01:24:48 G1 Young Generation GC in 219ms.  G1 Eden Space: 15401484288 -> 0; G1 Old
Gen: 4236918776 -> 5231503864; G1 Survivor Space: 2097152000 -
INFO  01:42:01 G1 Young Generation GC in 212ms.  G1 Eden Space: 16005464064 -> 0; G1 Old
Gen: 4320133112 -> 5100782576; G1 Survivor Space: 1627389952 -
INFO  01:48:28 G1 Young Generation GC in 209ms.  G1 Eden Space: 14730395648 -> 0; G1 Old
Gen: 8901994480 -> 9402393592; G1 Survivor Space: 1291845632 -
INFO  02:04:18 G1 Young Generation GC in 241ms.  G1 Eden Space: 19042140160 -> 0; G1 Old
Gen: 7825270264 -> 8327173624; G1 Survivor Space: 1560281088 -
INFO  02:08:09 Saved KeyCache (8634 items) in 94 ms
INFO  02:09:31 Redistributing index summaries
INFO  02:14:50 G1 Young Generation GC in 314ms.  G1 Eden Space: 19075694592 -> 0; G1 Old
Gen: 3505807352 -> 4140375528; G1 Survivor Space: 1526726656 -
INFO  02:48:35 G1 Young Generation GC in 321ms.  G1 Eden Space: 11576279040 -> 0; G1 Old
Gen: 3865890800 -> 4482588664; G1 Survivor Space: 1543503872 -
INFO  02:53:54 G1 Young Generation GC in 203ms.  G1 Eden Space: 17901289472 -> 0; G1 Old
Gen: 7927235056 -> 8552357368; G1 Survivor Space: 1593835520 -
INFO  03:04:33 G1 Young Generation GC in 345ms.  G1 Eden Space: 13505658880 -> 0; G1 Old
Gen: 3765058032 -> 4443454952; G1 Survivor Space: 1442840576 -
INFO  03:09:31 Redistributing index summaries
INFO  03:22:56 G1 Young Generation GC in 250ms.  G1 Eden Space: 18991808512 -> 0; G1 Old
Gen: 4806672880 -> 5340134904; G1 Survivor Space: 1610612736 -
INFO  03:35:13 G1 Young Generation GC in 210ms.  G1 Eden Space: 14445182976 -> 0; G1 Survivor
Space: 1073741824 -> 1744830464; 
INFO  03:38:42 G1 Young Generation GC in 214ms.  G1 Eden Space: 18303942656 -> 0; G1 Old
Gen: 4786540536 -> 5432116720; G1 Survivor Space: 1375731712 -
INFO  03:52:11 G1 Young Generation GC in 207ms.  G1 Eden Space: 117440512 -> 0; G1 Old
Gen: 12054831088 -> 10154180600; G1 Survivor Space: 1593835520 -
INFO  03:54:48 G1 Young Generation GC in 224ms.  G1 Eden Space: 16156459008 -> 0; G1 Old
Gen: 4303355896 -> 4801558520; G1 Survivor Space: 1560281088 -
INFO  04:09:31 Redistributing index summaries
INFO  04:16:09 G1 Young Generation GC in 216ms.  G1 Eden Space: 19142803456 -> 0; G1 Old
Gen: 5853311984 -> 6366188536; G1 Survivor Space: 1459617792 -
INFO  04:19:13 G1 Young Generation GC in 229ms.  G1 Eden Space: 19042140160 -> 0; G1 Old
Gen: 7826571256 -> 8457920504; G1 Survivor Space: 1560281088 -
INFO  04:27:41 G1 Young Generation GC in 311ms.  G1 Eden Space: 13723762688 -> 0; G1 Old
Gen: 14176747504 -> 14808230392; G1 Survivor Space: 1509949440
INFO  04:29:56 G1 Young Generation GC in 274ms.  G1 Eden Space: 18740150272 -> 0; G1 Old
Gen: 3376452080 -> 3818638328; G1 Survivor Space: 1459617792 -
INFO  04:33:40 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB
INFO  04:49:07 G1 Young Generation GC in 319ms.  G1 Eden Space: 16005464064 -> 0; G1 Old
Gen: 4435714040 -> 5023449072; G1 Survivor Space: 1442840576 -
INFO  04:54:06 G1 Young Generation GC in 208ms.  G1 Eden Space: 18706595840 -> 0; G1 Old
Gen: 7843348472 -> 8466889208; G1 Survivor Space: 1476395008 -
INFO  05:09:31 Redistributing index summaries


I found   INFO   Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB,
does it  mean the  memory is not enough ?


I want to know why  adding  a new node  spent  so  much time and what need I do to  complete
 a new node  joining the cluster ?


I need some help and really need some suggestions.



Thanks  a lot!





Mime
View raw message