cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "pavel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10371) Decommissioned nodes can remain in gossip
Date Thu, 18 Feb 2016 17:25:18 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15152664#comment-15152664
] 

pavel commented on CASSANDRA-10371:
-----------------------------------

Latest logs from 10.0.2.128 are: 

{noformat}
{"time":"2016-02-18T17:23:24.532Z","level":"TRACE","msg":"Adding state LOAD: 1.417907115E9"}
{"time":"2016-02-18T17:23:24.532Z","level":"TRACE","msg":"Sending a GossipDigestAck2Message
to /10.0.1.12"}
{"time":"2016-02-18T17:23:25.530Z","level":"TRACE","msg":"My heartbeat is now 3481162"}
{"time":"2016-02-18T17:23:25.530Z","level":"TRACE","msg":"Gossip Digests are : /10.0.2.168:1455284954:1611388
/10.0.2.128:1454668510:3481162 /10.0.1.129:1454674252:3463753 /10.0.1.88:1454668666:3480679
/10.0.1.12:1454675536:3459852 /10.0.2.117:1454676714:3456286 /10.0.1.32:1447859696:20667192
"}
{"time":"2016-02-18T17:23:25.530Z","level":"TRACE","msg":"Sending a GossipDigestSyn to /10.0.1.12
..."}
{"time":"2016-02-18T17:23:25.530Z","level":"TRACE","msg":"Performing status check ..."}
{"time":"2016-02-18T17:23:25.531Z","level":"TRACE","msg":"PHI for /10.0.1.129 : 0.9763986442368152"}
{"time":"2016-02-18T17:23:25.531Z","level":"TRACE","msg":"PHI for /10.0.2.168 : 1.0035592404405616"}
{"time":"2016-02-18T17:23:25.531Z","level":"TRACE","msg":"PHI for /10.0.1.12 : 0.9730840000367144"}
{"time":"2016-02-18T17:23:25.531Z","level":"TRACE","msg":"PHI for /10.0.2.117 : 0.9806746365769617"}
{"time":"2016-02-18T17:23:25.531Z","level":"TRACE","msg":"PHI for /10.0.1.88 : 0.9813799722457983"}
{"time":"2016-02-18T17:23:25.532Z","level":"TRACE","msg":"Received a GossipDigestAckMessage
from /10.0.1.12"}
{"time":"2016-02-18T17:23:25.532Z","level":"TRACE","msg":"Received ack with 2 digests and
4 states"}
{"time":"2016-02-18T17:23:25.532Z","level":"TRACE","msg":"reporting /10.0.1.88"}
{"time":"2016-02-18T17:23:25.532Z","level":"TRACE","msg":"reporting /10.0.1.129"}
{"time":"2016-02-18T17:23:25.532Z","level":"TRACE","msg":"reporting /10.0.1.12"}
{"time":"2016-02-18T17:23:25.532Z","level":"TRACE","msg":"reporting /10.0.2.117"}
{"time":"2016-02-18T17:23:25.532Z","level":"TRACE","msg":"/10.0.1.88local generation 1454668666,
remote generation 1454668666"}
{"time":"2016-02-18T17:23:25.532Z","level":"TRACE","msg":"Updating heartbeat state version
to 3480682 from 3480679 for /10.0.1.88 ..."}
{"time":"2016-02-18T17:23:25.532Z","level":"TRACE","msg":"/10.0.1.129local generation 1454674252,
remote generation 1454674252"}
{"time":"2016-02-18T17:23:25.532Z","level":"TRACE","msg":"Updating heartbeat state version
to 3463756 from 3463753 for /10.0.1.129 ..."}
{"time":"2016-02-18T17:23:25.532Z","level":"TRACE","msg":"/10.0.1.12local generation 1454675536,
remote generation 1454675536"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Updating heartbeat state version
to 3459855 from 3459852 for /10.0.1.12 ..."}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"/10.0.2.117local generation 1454676714,
remote generation 1454676714"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Updating heartbeat state version
to 3456289 from 3456286 for /10.0.2.117 ..."}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"local heartbeat version 3481162
greater than 3481159 for /10.0.2.128"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"local heartbeat version 20667192
greater than 0 for /10.0.1.32"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state RELEASE_VERSION: 2.1.11"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state HOST_ID: 88290568-e6ca-4674-a597-bad21e8a794c"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state STATUS: LEFT,-1183123795032699037,1454932471326"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state SEVERITY: 0.5167958736419678"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state SCHEMA: 612b0652-7783-384f-b412-a5288429ce5a"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state NET_VERSION: 8"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state DC: cassandra"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state INTERNAL_IP: 10.0.1.32"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state RACK: RAC1"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state RPC_ADDRESS: 10.10.65.1"}
?j??????7???6-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state TOKENS:???-G?????p?3{?_?=w?????d???E??)?J
6?
?4??????	?>?7? ?s?DQ??D?D?CAs1bs?u??27??01B??O?L????Ja|??q???<??*H???e??
[??????????B?I???	s???F???????D
?<?W'?????sj?s??b????Up??????????????$?'??????
4K????=??V&???f??N??5???'???@?               A?	????F????}???^????Q?-w?Z?d?=v??X??????YQ????
?N?N?????
}???L?[?R?1V??R?n?i??%?Rg???b?re}???????DS?'???C??'?????g?f,??
                                                              ?????)?e?G??.?Ga??~
                                                                                 ????(?~??E\m6????M????w_?p?????l???(8
??"I????N??????3?Yr?I%Q&?k??O???$???gw1??BVZ???z?????t??H?v??*R??[???l(????W??????/??]???????.???k????
                                                                                         
           ????!?????????-A?1?d?m? ?u??n?????"????.?*?
?R3)??Y2?????jT?pD??fRo???O?$3|?|??c??U??????X?H???	?????FL4???H,??w?????I??BBr?T??i0??Mq?8???u????x????B?]G??!c??6y???d38????6????.H??????K|?v?S???7????Q???l?Q??^?S??5?$?P
                                           ?{!?
?Vqh?k???1v?@?2                               ?
               H;eu/?Y????c??^-?w5?5S?X!? ?^??"/??zA"?<*#<?\?p
4?~fe?5??! ??c??:eY6??lO?u9!??P?9??I??9?cbJ?:/?]%??:????/;q??Z?C=?T?7=??{(xr>)????~86?3?Z>9????}z???R?x?>gH3J|?,?
R>bp????@6
??lD.?<gnF'???9V
               Gmv.!?gJm??J,<???6JO?~??K
                                        ??L????JL?Y8?5M???
                                                          ?*O?S??3P ??S?Pu'H?'Em?z(S??)U?T3???T&?$?YYU??,_OVAE???V?F???W???'?^W??????Y6>??t?Yb,l???ZzD??L?Y?E??\1??w(\u?)x2?\?A??)J^?Q
                                                                                         
                                                                                         
  ?_7l??_o?1G??`?????a5r??a ?p\???a?
44??    ?T
          ??c#?F?'c6???j?eWS???e??EUh>f???????
i???,%m?"?-k?iG;??B?in?k??i???
         {o??s??pk???lq?B??.r??L
                                s?????s?????Hs??Mn?`
?ti?A?t??                                          ????
?R?xUN?(Iy?G?@?y??+???zL[3>??z?J?oH?{????C?|??????}?b????L
/?3"}?                                                   ?)g)?|0
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state LOAD: 1.417907115E9"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Sending a GossipDigestAck2Message
to /10.0.1.12"}
{"time":"2016-02-18T17:23:26.531Z","level":"TRACE","msg":"My heartbeat is now 3481165"}
{"time":"2016-02-18T17:23:26.531Z","level":"TRACE","msg":"Gossip Digests are : /10.0.1.129:1454674252:3463756
/10.0.1.32:1447859696:20667192 /10.0.2.168:1455284954:1611388 /10.0.1.12:1454675536:3459855
/10.0.1.88:1454668666:3480682 /10.0.2.128:1454668510:3481165 /10.0.2.117:1454676714:3456289
"}
{"time":"2016-02-18T17:23:26.531Z","level":"TRACE","msg":"Sending a GossipDigestSyn to /10.0.1.88
..."}
{"time":"2016-02-18T17:23:26.531Z","level":"TRACE","msg":"Performing status check ..."}
{"time":"2016-02-18T17:23:26.531Z","level":"TRACE","msg":"PHI for /10.0.1.129 : 0.9761823849067737"}
{"time":"2016-02-18T17:23:26.531Z","level":"TRACE","msg":"PHI for /10.0.2.168 : 2.0081343786762047"}
{"time":"2016-02-18T17:23:26.531Z","level":"TRACE","msg":"PHI for /10.0.1.12 : 0.9725689805550068"}
{"time":"2016-02-18T17:23:26.531Z","level":"TRACE","msg":"PHI for /10.0.2.117 : 0.9793344483809482"}
{"time":"2016-02-18T17:23:26.531Z","level":"TRACE","msg":"PHI for /10.0.1.88 : 0.9800589923703654"}
{"time":"2016-02-18T17:23:26.532Z","level":"TRACE","msg":"Received a GossipDigestAckMessage
from /10.0.1.88"}
{"time":"2016-02-18T17:23:26.532Z","level":"TRACE","msg":"Received ack with 2 digests and
3 states"}
{"time":"2016-02-18T17:23:26.532Z","level":"TRACE","msg":"reporting /10.0.2.168"}
{"time":"2016-02-18T17:23:26.532Z","level":"TRACE","msg":"reporting /10.0.1.88"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"reporting /10.0.2.117"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"/10.0.2.168local generation 1455284954,
remote generation 1455284954"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Updating heartbeat state version
to 1611392 from 1611386 for /10.0.2.168 ..."}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"/10.0.1.88local generation 1454668666,
remote generation 1454668666"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Updating heartbeat state version
to 3480685 from 3480682 for /10.0.1.88 ..."}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"/10.0.2.117local generation 1454676714,
remote generation 1454676714"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Updating heartbeat state version
to 3456292 from 3456289 for /10.0.2.117 ..."}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"local heartbeat version 3481165
greater than 3481162 for /10.0.2.128"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"local heartbeat version 20667192
greater than 0 for /10.0.1.32"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Adding state RELEASE_VERSION: 2.1.11"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Adding state HOST_ID: 88290568-e6ca-4674-a597-bad21e8a794c"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Adding state STATUS: LEFT,-1183123795032699037,1454932471326"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Adding state SEVERITY: 0.5167958736419678"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Adding state SCHEMA: 612b0652-7783-384f-b412-a5288429ce5a"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Adding state NET_VERSION: 8"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Adding state DC: cassandra"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Adding state INTERNAL_IP: 10.0.1.32"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Adding state RACK: RAC1"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Adding state RPC_ADDRESS: 10.10.65.1"}
?j??????7???6-02-18T17:23:26.533Z","level":"TRACE","msg":"Adding state TOKENS:???-G?????p?3{?_?=w?????d???E??)?J
6?
?4??????	?>?7? ?s?DQ??D?D?CAs1bs?u??27??01B??O?L????Ja|??q???<??*H???e??
[??????????B?I???	s???F???????D
?<?W'?????sj?s??b????Up??????????????$?'??????
4K????=??V&???f??N??5???'???@?               A?	????F????}???^????Q?-w?Z?d?=v??X??????YQ????
?N?N?????
}???L?[?R?1V??R?n?i??%?Rg???b?re}???????DS?'???C??'?????g?f,??
                                                              ?????)?e?G??.?Ga??~
                                                                                 ????(?~??E\m6????M????w_?p?????l???(8
??"I????N??????3?Yr?I%Q&?k??O???$???gw1??BVZ???z?????t??H?v??*R??[???l(????W??????/??]???????.???k????
                                                                                         
           ????!?????????-A?1?d?m? ?u??n?????"????.?*?
?R3)??Y2?????jT?pD??fRo???O?$3|?|??c??U??????X?H???	?????FL4???H,??w?????I??BBr?T??i0??Mq?8???u????x????B?]G??!c??6y???d38????6????.H??????K|?v?S???7????Q???l?Q??^?S??5?$?P
                                           ?{!?
?Vqh?k???1v?@?2                               ?
               H;eu/?Y????c??^-?w5?5S?X!? ?^??"/??zA"?<*#<?\?p
4?~fe?5??! ??c??:eY6??lO?u9!??P?9??I??9?cbJ?:/?]%??:????/;q??Z?C=?T?7=??{(xr>)????~86?3?Z>9????}z???R?x?>gH3J|?,?
R>bp????@6
??lD.?<gnF'???9V
               Gmv.!?gJm??J,<???6JO?~??K
                                        ??L????JL?Y8?5M???
                                                          ?*O?S??3P ??S?Pu'H?'Em?z(S??)U?T3???T&?$?YYU??,_OVAE???V?F???W???'?^W??????Y6>??t?Yb,l???ZzD??L?Y?E??\1??w(\u?)x2?\?A??)J^?Q
                                                                                         
                                                                                         
  ?_7l??_o?1G??`?????a5r??a ?p\???a?
44??    ?T
          ??c#?F?'c6???j?eWS???e??EUh>f???????
i???,%m?"?-k?iG;??B?in?k??i???
         {o??s??pk???lq?B??.r??L
                                s?????s?????Hs??Mn?`
?ti?A?t??                                          ????
?R?xUN?(Iy?G?@?y??+???zL[3>??z?J?oH?{????C?|??????}?b????L
/?3"}?                                                   ?)g)?|0
{"time":"2016-02-18T17:23:26.534Z","level":"TRACE","msg":"Adding state LOAD: 1.417907115E9"}
{"time":"2016-02-18T17:23:26.534Z","level":"TRACE","msg":"Sending a GossipDigestAck2Message
to /10.0.1.88"}
{"time":"2016-02-18T17:23:27.441Z","level":"TRACE","msg":"Adding state SEVERITY: 0.25641027092933655"}
{"time":"2016-02-18T17:23:27.442Z","level":"TRACE","msg":"Received a GossipDigestAck2Message
from /10.0.1.88"}
{"time":"2016-02-18T17:23:27.442Z","level":"TRACE","msg":"reporting /10.0.2.168"}
{"time":"2016-02-18T17:23:27.442Z","level":"TRACE","msg":"reporting /10.0.1.88"}
{"time":"2016-02-18T17:23:27.442Z","level":"TRACE","msg":"reporting /10.0.1.129"}
{"time":"2016-02-18T17:23:27.443Z","level":"TRACE","msg":"reporting /10.0.1.12"}
{"time":"2016-02-18T17:23:27.443Z","level":"TRACE","msg":"/10.0.2.168local generation 1455284954,
remote generation 1455284954"}
{"time":"2016-02-18T17:23:27.443Z","level":"TRACE","msg":"Updating heartbeat state version
to 1611395 from 1611392 for /10.0.2.168 ..."}
{"time":"2016-02-18T17:23:27.443Z","level":"TRACE","msg":"/10.0.1.88local generation 1454668666,
remote generation 1454668666"}
{"time":"2016-02-18T17:23:27.443Z","level":"TRACE","msg":"Updating heartbeat state version
to 3480688 from 3480685 for /10.0.1.88 ..."}
{"time":"2016-02-18T17:23:27.443Z","level":"TRACE","msg":"/10.0.1.129local generation 1454674252,
remote generation 1454674252"}
{"time":"2016-02-18T17:23:27.443Z","level":"TRACE","msg":"Updating heartbeat state version
to 3463762 from 3463756 for /10.0.1.129 ..."}
{"time":"2016-02-18T17:23:27.443Z","level":"TRACE","msg":"/10.0.1.12local generation 1454675536,
remote generation 1454675536"}
{"time":"2016-02-18T17:23:27.443Z","level":"TRACE","msg":"Updating heartbeat state version
to 3459858 from 3459855 for /10.0.1.12 ..."}
{"time":"2016-02-18T17:23:27.531Z","level":"TRACE","msg":"My heartbeat is now 3481168"}
{"time":"2016-02-18T17:23:27.531Z","level":"TRACE","msg":"Gossip Digests are : /10.0.2.128:1454668510:3481168
/10.0.1.32:1447859696:20667192 /10.0.1.88:1454668666:3480688 /10.0.2.168:1455284954:1611395
/10.0.1.129:1454674252:3463762 /10.0.2.117:1454676714:3456294 /10.0.1.12:1454675536:3459858
"}
{"time":"2016-02-18T17:23:27.531Z","level":"TRACE","msg":"Sending a GossipDigestSyn to /10.0.1.129
..."}
{"time":"2016-02-18T17:23:27.531Z","level":"TRACE","msg":"Performing status check ..."}
{"time":"2016-02-18T17:23:27.532Z","level":"TRACE","msg":"PHI for /10.0.1.129 : 0.08694391920660251"}
{"time":"2016-02-18T17:23:27.532Z","level":"TRACE","msg":"PHI for /10.0.2.168 : 0.08943066263688543"}
{"time":"2016-02-18T17:23:27.532Z","level":"TRACE","msg":"PHI for /10.0.1.12 : 0.08651621637584873"}
{"time":"2016-02-18T17:23:27.532Z","level":"TRACE","msg":"PHI for /10.0.2.117 : 0.9803039634425629"}
{"time":"2016-02-18T17:23:27.532Z","level":"TRACE","msg":"PHI for /10.0.1.88 : 0.08748333029792915"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Received a GossipDigestAckMessage
from /10.0.1.129"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Received ack with 3 digests and
0 states"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"local heartbeat version 3480688
greater than 3480685 for /10.0.1.88"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"local heartbeat version 3481168
greater than 3481165 for /10.0.2.128"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Adding state SEVERITY: 0.25641027092933655"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"local heartbeat version 20667192
greater than 0 for /10.0.1.32"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Adding state RELEASE_VERSION: 2.1.11"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Adding state HOST_ID: 88290568-e6ca-4674-a597-bad21e8a794c"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Adding state STATUS: LEFT,-1183123795032699037,1454932471326"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Adding state SEVERITY: 0.5167958736419678"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Adding state SCHEMA: 612b0652-7783-384f-b412-a5288429ce5a"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Adding state NET_VERSION: 8"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Adding state DC: cassandra"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Adding state INTERNAL_IP: 10.0.1.32"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Adding state RACK: RAC1"}
{"time":"2016-02-18T17:23:27.534Z","level":"TRACE","msg":"Adding state RPC_ADDRESS: 10.10.65.1"}
?j??????7???6-02-18T17:23:27.534Z","level":"TRACE","msg":"Adding state TOKENS:???-G?????p?3{?_?=w?????d???E??)?J
6?
?4??????	?>?7? ?s?DQ??D?D?CAs1bs?u??27??01B??O?L????Ja|??q???<??*H???e??
[??????????B?I???	s???F???????D
?<?W'?????sj?s??b????Up??????????????$?'??????
4K????=??V&???f??N??5???'???@?               A?	????F????}???^????Q?-w?Z?d?=v??X??????YQ????
?N?N?????
}???L?[?R?1V??R?n?i??%?Rg???b?re}???????DS?'???C??'?????g?f,??
                                                              ?????)?e?G??.?Ga??~
                                                                                 ????(?~??E\m6????M????w_?p?????l???(8
??"I????N??????3?Yr?I%Q&?k??O???$???gw1??BVZ???z?????t??H?v??*R??[???l(????W??????/??]???????.???k????
                                                                                         
           ????!?????????-A?1?d?m? ?u??n?????"????.?*?
?R3)??Y2?????jT?pD??fRo???O?$3|?|??c??U??????X?H???	?????FL4???H,??w?????I??BBr?T??i0??Mq?8???u????x????B?]G??!c??6y???d38????6????.H??????K|?v?S???7????Q???l?Q??^?S??5?$?P
                                           ?{!?
?Vqh?k???1v?@?2                               ?
               H;eu/?Y????c??^-?w5?5S?X!? ?^??"/??zA"?<*#<?\?p
4?~fe?5??! ??c??:eY6??lO?u9!??P?9??I??9?cbJ?:/?]%??:????/;q??Z?C=?T?7=??{(xr>)????~86?3?Z>9????}z???R?x?>gH3J|?,?
R>bp????@6
??lD.?<gnF'???9V
               Gmv.!?gJm??J,<???6JO?~??K
                                        ??L????JL?Y8?5M???
                                                          ?*O?S??3P ??S?Pu'H?'Em?z(S??)U?T3???T&?$?YYU??,_OVAE???V?F???W???'?^W??????Y6>??t?Yb,l???ZzD??L?Y?E??\1??w(\u?)x2?\?A??)J^?Q
                                                                                         
                                                                                         
  ?_7l??_o?1G??`?????a5r??a ?p\???a?
44??    ?T
          ??c#?F?'c6???j?eWS???e??EUh>f???????
i???,%m?"?-k?iG;??B?in?k??i???
         {o??s??pk???lq?B??.r??L
                                s?????s?????Hs??Mn?`
?ti?A?t??                                          ????
?R?xUN?(Iy?G?@?y??+???zL[3>??z?J?oH?{????C?|??????}?b????L
/?3"}?                                                   ?)g)?|0
{"time":"2016-02-18T17:23:27.534Z","level":"TRACE","msg":"Adding state LOAD: 1.417907115E9"}
{"time":"2016-02-18T17:23:27.534Z","level":"TRACE","msg":"Sending a GossipDigestAck2Message
to /10.0.1.129"}
{"time":"2016-02-18T17:23:28.532Z","level":"TRACE","msg":"My heartbeat is now 3481171"}
{"time":"2016-02-18T17:23:28.532Z","level":"TRACE","msg":"Gossip Digests are : /10.0.1.88:1454668666:3480688
/10.0.2.117:1454676714:3456294 /10.0.2.168:1455284954:1611395 /10.0.2.128:1454668510:3481171
/10.0.1.129:1454674252:3463762 /10.0.1.12:1454675536:3459858 /10.0.1.32:1447859696:20667192
"}
{"time":"2016-02-18T17:23:28.532Z","level":"TRACE","msg":"Sending a GossipDigestSyn to /10.0.2.117
..."}
{"time":"2016-02-18T17:23:28.532Z","level":"TRACE","msg":"Performing status check ..."}
{"time":"2016-02-18T17:23:28.532Z","level":"TRACE","msg":"PHI for /10.0.1.129 : 1.0647911374060737"}
{"time":"2016-02-18T17:23:28.532Z","level":"TRACE","msg":"PHI for /10.0.2.168 : 1.0934171825795906"}
{"time":"2016-02-18T17:23:28.532Z","level":"TRACE","msg":"PHI for /10.0.1.12 : 1.0591266224847133"}
{"time":"2016-02-18T17:23:28.532Z","level":"TRACE","msg":"PHI for /10.0.2.117 : 1.962043900044831"}
{"time":"2016-02-18T17:23:28.532Z","level":"TRACE","msg":"PHI for /10.0.1.88 : 1.069004979529513"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Received a GossipDigestAckMessage
from /10.0.2.117"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Received ack with 2 digests and
5 states"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"reporting /10.0.2.168"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"reporting /10.0.1.88"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"reporting /10.0.1.129"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"reporting /10.0.1.12"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"reporting /10.0.2.117"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"/10.0.2.168local generation 1455284954,
remote generation 1455284954"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Updating heartbeat state version
to 1611398 from 1611395 for /10.0.2.168 ..."}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"/10.0.1.88local generation 1454668666,
remote generation 1454668666"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Updating heartbeat state version
to 3480691 from 3480688 for /10.0.1.88 ..."}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"/10.0.1.129local generation 1454674252,
remote generation 1454674252"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Updating heartbeat state version
to 3463765 from 3463762 for /10.0.1.129 ..."}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"/10.0.1.12local generation 1454675536,
remote generation 1454675536"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Updating heartbeat state version
to 3459864 from 3459858 for /10.0.1.12 ..."}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"/10.0.2.117local generation 1454676714,
remote generation 1454676714"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Updating heartbeat state version
to 3456298 from 3456292 for /10.0.2.117 ..."}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"local heartbeat version 3481171
greater than 3481168 for /10.0.2.128"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"local heartbeat version 20667192
greater than 0 for /10.0.1.32"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Adding state RELEASE_VERSION: 2.1.11"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Adding state HOST_ID: 88290568-e6ca-4674-a597-bad21e8a794c"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Adding state STATUS: LEFT,-1183123795032699037,1454932471326"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Adding state SEVERITY: 0.5167958736419678"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Adding state SCHEMA: 612b0652-7783-384f-b412-a5288429ce5a"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Adding state NET_VERSION: 8"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Adding state DC: cassandra"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Adding state INTERNAL_IP: 10.0.1.32"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Adding state RACK: RAC1"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Adding state RPC_ADDRESS: 10.10.65.1"}
?j??????7???6-02-18T17:23:28.533Z","level":"TRACE","msg":"Adding state TOKENS:???-G?????p?3{?_?=w?????d???E??)?J
6?
?4??????	?>?7? ?s?DQ??D?D?CAs1bs?u??27??01B??O?L????Ja|??q???<??*H???e??
[??????????B?I???	s???F???????D
?<?W'?????sj?s??b????Up??????????????$?'??????
4K????=??V&???f??N??5???'???@?               A?	????F????}???^????Q?-w?Z?d?=v??X??????YQ????
?N?N?????
}???L?[?R?1V??R?n?i??%?Rg???b?re}???????DS?'???C??'?????g?f,??
                                                              ?????)?e?G??.?Ga??~
                                                                                 ????(?~??E\m6????M????w_?p?????l???(8
??"I????N??????3?Yr?I%Q&?k??O???$???gw1??BVZ???z?????t??H?v??*R??[???l(????W??????/??]???????.???k????
                                                                                         
           ????!?????????-A?1?d?m? ?u??n?????"????.?*?
?R3)??Y2?????jT?pD??fRo???O?$3|?|??c??U??????X?H???	?????FL4???H,??w?????I??BBr?T??i0??Mq?8???u????x????B?]G??!c??6y???d38????6????.H??????K|?v?S???7????Q???l?Q??^?S??5?$?P
                                           ?{!?
?Vqh?k???1v?@?2                               ?
               H;eu/?Y????c??^-?w5?5S?X!? ?^??"/??zA"?<*#<?\?p
4?~fe?5??! ??c??:eY6??lO?u9!??P?9??I??9?cbJ?:/?]%??:????/;q??Z?C=?T?7=??{(xr>)????~86?3?Z>9????}z???R?x?>gH3J|?,?
R>bp????@6
??lD.?<gnF'???9V
               Gmv.!?gJm??J,<???6JO?~??K
                                        ??L????JL?Y8?5M???
                                                          ?*O?S??3P ??S?Pu'H?'Em?z(S??)U?T3???T&?$?YYU??,_OVAE???V?F???W???'?^W??????Y6>??t?Yb,l???ZzD??L?Y?E??\1??w(\u?)x2?\?A??)J^?Q
                                                                                         
                                                                                         
  ?_7l??_o?1G??`?????a5r??a ?p\???a?
44??    ?T
          ??c#?F?'c6???j?eWS???e??EUh>f???????
i???,%m?"?-k?iG;??B?in?k??i???
         {o??s??pk???lq?B??.r??L
                                s?????s?????Hs??Mn?`
?ti?A?t??                                          ????
?R?xUN?(Iy?G?@?y??+???zL[3>??z?J?oH?{????C?|??????}?b????L
/?3"}?                                                   ?)g)?|0
{noformat}

> Decommissioned nodes can remain in gossip
> -----------------------------------------
>
>                 Key: CASSANDRA-10371
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10371
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Distributed Metadata
>            Reporter: Brandon Williams
>            Assignee: Joel Knighton
>            Priority: Minor
>
> This may apply to other dead states as well.  Dead states should be expired after 3 days.
 In the case of decom we attach a timestamp to let the other nodes know when it should be
expired.  It has been observed that sometimes a subset of nodes in the cluster never expire
the state, and through heap analysis of these nodes it is revealed that the epstate.isAlive
check returns true when it should return false, which would allow the state to be evicted.
 This may have been affected by CASSANDRA-8336.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message