cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "pavel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10371) Decommissioned nodes can remain in gossip
Date Tue, 16 Feb 2016 16:50:18 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15148893#comment-15148893
] 

pavel commented on CASSANDRA-10371:
-----------------------------------

Logs. Dead node is 10.0.1.32

{noformat}
{"time":"2016-02-16T16:31:24.209Z","level":"TRACE","msg":"requestAll for /10.0.1.32"}
{"time":"2016-02-16T16:31:24.210Z","level":"TRACE","msg":"Received a GossipDigestAck2Message
from /10.0.2.128"}
{"time":"2016-02-16T16:31:24.210Z","level":"TRACE","msg":"reporting /10.0.2.128"}
{"time":"2016-02-16T16:31:24.210Z","level":"TRACE","msg":"/10.0.2.128local generation 1454668510,
remote generation 1454668510"}
{"time":"2016-02-16T16:31:24.210Z","level":"TRACE","msg":"Updating heartbeat state version
to 2947566 from 2947563 for /10.0.2.128 ..."}
{"time":"2016-02-16T16:31:24.210Z","level":"TRACE","msg":"Ignoring gossip for /10.0.1.32 because
it is quarantined"}
{"time":"2016-02-16T16:31:24.351Z","level":"TRACE","msg":"My heartbeat is now 2947086"}
{"time":"2016-02-16T16:31:24.351Z","level":"TRACE","msg":"Gossip Digests are : /10.0.1.88:1454668666:2947086
/10.0.1.129:1454674252:2930159 /10.0.2.128:1454668510:2947566 /10.0.1.12:1454675536:2926261
/10.0.2.117:1454676714:2922695 /10.0.2.168:1455284954:1077795 "}
{"time":"2016-02-16T16:31:24.351Z","level":"TRACE","msg":"Sending a GossipDigestSyn to /10.0.1.12
..."}
{"time":"2016-02-16T16:31:24.351Z","level":"TRACE","msg":"Performing status check ..."}
{"time":"2016-02-16T16:31:24.351Z","level":"TRACE","msg":"PHI for /10.0.2.128 : 0.13901063851144269"}
{"time":"2016-02-16T16:31:24.351Z","level":"TRACE","msg":"PHI for /10.0.1.129 : 0.21683730070438223"}
{"time":"2016-02-16T16:31:24.351Z","level":"TRACE","msg":"PHI for /10.0.2.168 : 0.2206211272280096"}
{"time":"2016-02-16T16:31:24.351Z","level":"TRACE","msg":"PHI for /10.0.1.12 : 0.2149248608862009"}
{"time":"2016-02-16T16:31:24.352Z","level":"TRACE","msg":"PHI for /10.0.2.117 : 0.2130329987137985"}
{"time":"2016-02-16T16:31:24.352Z","level":"DEBUG","msg":"60000 elapsed, /10.0.1.32 gossip
quarantine over"}
{"time":"2016-02-16T16:31:24.352Z","level":"TRACE","msg":"Received a GossipDigestAckMessage
from /10.0.1.12"}
{"time":"2016-02-16T16:31:24.352Z","level":"TRACE","msg":"Received ack with 2 digests and
0 states"}
{"time":"2016-02-16T16:31:24.352Z","level":"TRACE","msg":"local heartbeat version 2947086
greater than 2947085 for /10.0.1.88"}
{"time":"2016-02-16T16:31:24.352Z","level":"TRACE","msg":"local heartbeat version 2947566
greater than 2947565 for /10.0.2.128"}
{"time":"2016-02-16T16:31:24.352Z","level":"TRACE","msg":"Sending a GossipDigestAck2Message
to /10.0.1.12"}
{"time":"2016-02-16T16:31:24.542Z","level":"TRACE","msg":"local heartbeat version 2926261
greater than 2926258 for /10.0.1.12"}
{"time":"2016-02-16T16:31:24.542Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:24.542Z","level":"TRACE","msg":"local heartbeat version 2947086
greater than 2947080 for /10.0.1.88"}
{"time":"2016-02-16T16:31:24.542Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:24.542Z","level":"TRACE","msg":"local heartbeat version 2947566
greater than 2947563 for /10.0.2.128"}
{"time":"2016-02-16T16:31:24.542Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:24.543Z","level":"TRACE","msg":"Received a GossipDigestAck2Message
from /10.0.1.129"}
{"time":"2016-02-16T16:31:24.543Z","level":"TRACE","msg":"reporting /10.0.1.129"}
{"time":"2016-02-16T16:31:24.543Z","level":"TRACE","msg":"/10.0.1.129local generation 1454674252,
remote generation 1454674252"}
{"time":"2016-02-16T16:31:24.543Z","level":"TRACE","msg":"Updating heartbeat state version
to 2930162 from 2930159 for /10.0.1.129 ..."}
{"time":"2016-02-16T16:31:25.209Z","level":"TRACE","msg":"local heartbeat version 2930162
greater than 2930159 for /10.0.1.129"}
{"time":"2016-02-16T16:31:25.209Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:25.209Z","level":"TRACE","msg":"local heartbeat version 2947086
greater than 2947085 for /10.0.1.88"}
{"time":"2016-02-16T16:31:25.209Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:25.209Z","level":"TRACE","msg":"requestAll for /10.0.1.32"}
{"time":"2016-02-16T16:31:25.210Z","level":"TRACE","msg":"Received a GossipDigestAck2Message
from /10.0.2.128"}
{"time":"2016-02-16T16:31:25.210Z","level":"TRACE","msg":"reporting /10.0.2.128"}
{"time":"2016-02-16T16:31:25.210Z","level":"TRACE","msg":"reporting /10.0.2.117"}
{"time":"2016-02-16T16:31:25.210Z","level":"TRACE","msg":"/10.0.2.128local generation 1454668510,
remote generation 1454668510"}
{"time":"2016-02-16T16:31:25.210Z","level":"TRACE","msg":"Updating heartbeat state version
to 2947569 from 2947566 for /10.0.2.128 ..."}
{"time":"2016-02-16T16:31:25.210Z","level":"TRACE","msg":"reporting /10.0.1.32"}
{"time":"2016-02-16T16:31:25.210Z","level":"TRACE","msg":"Adding endpoint state for /10.0.1.32"}
{"time":"2016-02-16T16:31:25.210Z","level":"DEBUG","msg":"Not marking /10.0.1.32 alive due
to dead state"}
{"time":"2016-02-16T16:31:25.211Z","level":"TRACE","msg":"marking as down /10.0.1.32"}
{"time":"2016-02-16T16:31:25.211Z","level":"INFO","msg":"InetAddress /10.0.1.32 is now DOWN"}
{"time":"2016-02-16T16:31:25.211Z","level":"TRACE","msg":"Notified [org.apache.cassandra.service.StorageService@2a92d5f9,
org.apache.cassandra.locator.ReconnectableSnitchHelper@63e5499, org.apache.cassandra.service.LoadBroadcaster@7a3e4f9f]"}
{"time":"2016-02-16T16:31:25.211Z","level":"DEBUG","msg":"Ignoring state change for dead or
unknown endpoint: /10.0.1.32"}
{"time":"2016-02-16T16:31:25.211Z","level":"DEBUG","msg":"Ignoring state change for dead or
unknown endpoint: /10.0.1.32"}
{"time":"2016-02-16T16:31:25.211Z","level":"DEBUG","msg":"Ignoring state change for dead or
unknown endpoint: /10.0.1.32"}
{"time":"2016-02-16T16:31:25.211Z","level":"DEBUG","msg":"Ignoring state change for dead or
unknown endpoint: /10.0.1.32"}
{"time":"2016-02-16T16:31:25.211Z","level":"DEBUG","msg":"Ignoring state change for dead or
unknown endpoint: /10.0.1.32"}
{"time":"2016-02-16T16:31:25.211Z","level":"DEBUG","msg":"Ignoring state change for dead or
unknown endpoint: /10.0.1.32"}
{"time":"2016-02-16T16:31:25.211Z","level":"DEBUG","msg":"Ignoring state change for dead or
unknown endpoint: /10.0.1.32"}
{"time":"2016-02-16T16:31:25.211Z","level":"DEBUG","msg":"Ignoring state change for dead or
unknown endpoint: /10.0.1.32"}
{"time":"2016-02-16T16:31:25.212Z","level":"DEBUG","msg":"Ignoring state change for dead or
unknown endpoint: /10.0.1.32"}
{"time":"2016-02-16T16:31:25.212Z","level":"DEBUG","msg":"Ignoring state change for dead or
unknown endpoint: /10.0.1.32"}
{"time":"2016-02-16T16:31:25.212Z","level":"DEBUG","msg":"Node /10.0.1.32 state left, tokens
[-1183123795032699037, -1237172198872744999, -1270149265263593564, -1389400073696907418, -1402231454860468978,
-1420312602012480180, -1627335600868165275, -1713896876933182948, -1714549883939918780, -1741474774932965377,
-1792210053935902256, -1807601543024917385, -1854566658540932161, -1912378675308644559, -1963516450452847956,
-2081077201955154820, -2099259848054237265, -2129680696766320789, -2221883494221453885, -2310616694711668360,
-2331548641412448139, -2334622440252028474, -2411111763982847025, -245865888987490824, -246385138307167507,
-2473543956019538623, -2490169864633659751, -2532042510174406079, -2540038371895843059, -2577021784711590342,
-2593326939593610701, -2705600331335690795, -2770464252902532656, -2839608713233822191, -2912475620975565625,
-2943366603912110567, -2949295810903492735, -2954629576212069889, -3095752208331422097, -3121066151673234082,
-3185818042633507184, -3219579857569752331, -3286343234413829618, -3289689335761352230, -3361565551667832402,
-3383031027800371008, -3412816001810159961, -3457683958466098497, -3508695836421455440, -3532532777067013075,
-3543699029518234044, -3580715661159129955, -3599252547477456660, -3661459590777641370, -3667538041993955650,
-3690186824731493069, -3814115795769795724, -409080988277749690, -4156007243077529438, -4168774893038102265,
-4213854018229603756, -4249034660629583574, -4281296366845033280, -4320926472067799077, -4545355489460541553,
-4630628418422675470, -4654983718222492637, -4768547837932983920, -5159507483120816494, -51965338274886423,
-5199723923113410828, -5223817829501441469, -5267064983314818217, -5348036139422617554, -5364643938379720849,
-5495473223858010026, -5538859875930354329, -554259440667407064, -5547533646691145209, -5599903631185515707,
-5779719603131964183, -5932457424237549800, -6205324647087801173, -6229187725024886147, -6231270541967013773,
-6235596621005793282, -6243196665962158041, -6264073286569207959, -6427783773364474000, -6501124205004483449,
-6515615874324067226, -6573166604041811188, -6637220677879215528, -6651782973192407164, -6660806062319128258,
-6745026564913028874, -6812331251174808386, -7084494716757599690, -7149011211591734620, -7274798955118474541,
-731652465373161574, -7403663263352230504, -7447942789148217722, -7449824614964680985, -7587057746419387993,
-7641766861615411515, -7685202236673643074, -7707429459189222115, -7742454891260223055, -7833962528790251697,
-7931725553821464164, -7939773289897241357, -8098012638292660116, -8219265643204532541, -8235265212011847570,
-8252620209307958750, -8332424581430025253, -8349765834148048802, -836487426771203231, -8505914076204635715,
-851663313261657661, -8519826107865925909, -8695372492829159498, -8942575831111517062, -8976797677584625277,
-9051099539809741903, -963992846195142625, 1003830146651642498, 1178027849963644975, 1224976256801927215,
1244352078631038965, 1417164721674217220, 1451341591539912399, 1491082438042287951, 1514848536642096395,
1779901662209221643, 1854090933062166699, 1943140502461231414, 1994380979939543148, 2007534562233247106,
217709005954998613, 2235989942794609624, 2241300505721899121, 2246701380267263353, 231183097291070161,
2426913079401922388, 2451699806267654588, 2506037868809622944, 2539177368085335242, 2585691867824138041,
2615916489656464006, 2629622650576085045, 2657709892891143999, 269267015073834956, 2978608649954009655,
2994749269564046800, 3002378570488766614, 3121394566209939036, 3141180417712080576, 3154166401188000671,
3295643071378191280, 3367253256599281972, 3373165743103532333, 3410319489097671220, 341173714064808701,
35319237546365581, 358168701228763255, 3695798865479120909, 3784007090181772945, 3859215587032554422,
388414320421460408, 3958884439975359895, 4116791292047135794, 4157923384507172767, 4177479403175856644,
4192843956567907187, 4237835213950914050, 4283455088657318777, 4437755738583404917, 4464954521637384793,
4479350931740982844, 448168435071015674, 4483893029099418629, 4495279466361191724, 4626894508809678069,
4913067912592229158, 5055224510936602124, 5146899889661568824, 5333227006937586923, 5344713819757688419,
5354664536118782383, 5407856766112305920, 5524147618224097934, 5530799914723849578, 5594217902784391863,
5740424123774088150, 5773813700926694122, 5797583289244413653, 596003237801815323, 6051342740892152639,
6053457919557951562, 6063733117952809262, 6188792405330710479, 6215325692940778409, 6255014464234787541,
6307265543356112469, 6331411861541807584, 6428394292228979654, 6440759262443849168, 6519598944774579329,
6559591672744555181, 6637797466881599715, 6662404025200321225, 6668777379548711499, 6828109123001551836,
6861072227483193626, 6876892389046410599, 6982428538201939129, 7004630263067699041, 703529510861716420,
7035478043169974728, 713068522342900631, 7143695289303246630, 7149047549904215441, 7302396691607427712,
7315761441890909840, 7410347447605570455, 754235745869667971, 7544573809574793144, 7586097788690287546,
7597038064500920869, 7614889355576709597, 7624148922826428440, 7627724472890172915, 7896536432019982357,
8070449930340402428, 8100741102346464407, 8206476157175887120, 8277001814877694995, 8323125797400536081,
8331290716255307824, 8338939509697372178, 834182466138639995, 8388253326997854443, 8414277759665359090,
8670923111306643765, 8769994749009616915, 8770706585661243637, 8812518846508548955, 8847966629758817746,
8906298870427014138, 8987438727642595319, 9050092026849126476, 933979495781858352, 962613924858074837,
985754492835172659]"}
{"time":"2016-02-16T16:31:25.212Z","level":"DEBUG","msg":"adding expire time for endpoint
: /10.0.1.32 (1454932471326)"}
{"time":"2016-02-16T16:31:25.212Z","level":"INFO","msg":"Removing tokens [-1183123795032699037,
-1237172198872744999, -1270149265263593564, -1389400073696907418, -1402231454860468978, -1420312602012480180,
-1627335600868165275, -1713896876933182948, -1714549883939918780, -1741474774932965377, -1792210053935902256,
-1807601543024917385, -1854566658540932161, -1912378675308644559, -1963516450452847956, -2081077201955154820,
-2099259848054237265, -2129680696766320789, -2221883494221453885, -2310616694711668360, -2331548641412448139,
-2334622440252028474, -2411111763982847025, -245865888987490824, -246385138307167507, -2473543956019538623,
-2490169864633659751, -2532042510174406079, -2540038371895843059, -2577021784711590342, -2593326939593610701,
-2705600331335690795, -2770464252902532656, -2839608713233822191, -2912475620975565625, -2943366603912110567,
-2949295810903492735, -2954629576212069889, -3095752208331422097, -3121066151673234082, -3185818042633507184,
-3219579857569752331, -3286343234413829618, -3289689335761352230, -3361565551667832402, -3383031027800371008,
-3412816001810159961, -3457683958466098497, -3508695836421455440, -3532532777067013075, -3543699029518234044,
-3580715661159129955, -3599252547477456660, -3661459590777641370, -3667538041993955650, -3690186824731493069,
-3814115795769795724, -409080988277749690, -4156007243077529438, -4168774893038102265, -4213854018229603756,
-4249034660629583574, -4281296366845033280, -4320926472067799077, -4545355489460541553, -4630628418422675470,
-4654983718222492637, -4768547837932983920, -5159507483120816494, -51965338274886423, -5199723923113410828,
-5223817829501441469, -5267064983314818217, -5348036139422617554, -5364643938379720849, -5495473223858010026,
-5538859875930354329, -554259440667407064, -5547533646691145209, -5599903631185515707, -5779719603131964183,
-5932457424237549800, -6205324647087801173, -6229187725024886147, -6231270541967013773, -6235596621005793282,
-6243196665962158041, -6264073286569207959, -6427783773364474000, -6501124205004483449, -6515615874324067226,
-6573166604041811188, -6637220677879215528, -6651782973192407164, -6660806062319128258, -6745026564913028874,
-6812331251174808386, -7084494716757599690, -7149011211591734620, -7274798955118474541, -731652465373161574,
-7403663263352230504, -7447942789148217722, -7449824614964680985, -7587057746419387993, -7641766861615411515,
-7685202236673643074, -7707429459189222115, -7742454891260223055, -7833962528790251697, -7931725553821464164,
-7939773289897241357, -8098012638292660116, -8219265643204532541, -8235265212011847570, -8252620209307958750,
-8332424581430025253, -8349765834148048802, -836487426771203231, -8505914076204635715, -851663313261657661,
-8519826107865925909, -8695372492829159498, -8942575831111517062, -8976797677584625277, -9051099539809741903,
-963992846195142625, 1003830146651642498, 1178027849963644975, 1224976256801927215, 1244352078631038965,
1417164721674217220, 1451341591539912399, 1491082438042287951, 1514848536642096395, 1779901662209221643,
1854090933062166699, 1943140502461231414, 1994380979939543148, 2007534562233247106, 217709005954998613,
2235989942794609624, 2241300505721899121, 2246701380267263353, 231183097291070161, 2426913079401922388,
2451699806267654588, 2506037868809622944, 2539177368085335242, 2585691867824138041, 2615916489656464006,
2629622650576085045, 2657709892891143999, 269267015073834956, 2978608649954009655, 2994749269564046800,
3002378570488766614, 3121394566209939036, 3141180417712080576, 3154166401188000671, 3295643071378191280,
3367253256599281972, 3373165743103532333, 3410319489097671220, 341173714064808701, 35319237546365581,
358168701228763255, 3695798865479120909, 3784007090181772945, 3859215587032554422, 388414320421460408,
3958884439975359895, 4116791292047135794, 4157923384507172767, 4177479403175856644, 4192843956567907187,
4237835213950914050, 4283455088657318777, 4437755738583404917, 4464954521637384793, 4479350931740982844,
448168435071015674, 4483893029099418629, 4495279466361191724, 4626894508809678069, 4913067912592229158,
5055224510936602124, 5146899889661568824, 5333227006937586923, 5344713819757688419, 5354664536118782383,
5407856766112305920, 5524147618224097934, 5530799914723849578, 5594217902784391863, 5740424123774088150,
5773813700926694122, 5797583289244413653, 596003237801815323, 6051342740892152639, 6053457919557951562,
6063733117952809262, 6188792405330710479, 6215325692940778409, 6255014464234787541, 6307265543356112469,
6331411861541807584, 6428394292228979654, 6440759262443849168, 6519598944774579329, 6559591672744555181,
6637797466881599715, 6662404025200321225, 6668777379548711499, 6828109123001551836, 6861072227483193626,
6876892389046410599, 6982428538201939129, 7004630263067699041, 703529510861716420, 7035478043169974728,
713068522342900631, 7143695289303246630, 7149047549904215441, 7302396691607427712, 7315761441890909840,
7410347447605570455, 754235745869667971, 7544573809574793144, 7586097788690287546, 7597038064500920869,
7614889355576709597, 7624148922826428440, 7627724472890172915, 7896536432019982357, 8070449930340402428,
8100741102346464407, 8206476157175887120, 8277001814877694995, 8323125797400536081, 8331290716255307824,
8338939509697372178, 834182466138639995, 8388253326997854443, 8414277759665359090, 8670923111306643765,
8769994749009616915, 8770706585661243637, 8812518846508548955, 8847966629758817746, 8906298870427014138,
8987438727642595319, 9050092026849126476, 933979495781858352, 962613924858074837, 985754492835172659]
for /10.0.1.32"}
{"time":"2016-02-16T16:31:25.218Z","level":"DEBUG","msg":"removing endpoint /10.0.1.32"}
{"time":"2016-02-16T16:31:25.218Z","level":"DEBUG","msg":"Ignoring state change for dead or
unknown endpoint: /10.0.1.32"}
{"time":"2016-02-16T16:31:25.221Z","level":"TRACE","msg":"/10.0.2.168local generation 1455284954,
remote generation 1455284954"}
{"time":"2016-02-16T16:31:25.221Z","level":"TRACE","msg":"Updating heartbeat state version
to 1077795 from 1077795 for /10.0.2.168 ..."}
{"time":"2016-02-16T16:31:25.221Z","level":"TRACE","msg":"/10.0.2.117local generation 1454676714,
remote generation 1454676714"}
{"time":"2016-02-16T16:31:25.221Z","level":"TRACE","msg":"Updating heartbeat state version
to 2922701 from 2922695 for /10.0.2.117 ..."}
{"time":"2016-02-16T16:31:25.352Z","level":"TRACE","msg":"My heartbeat is now 2947089"}
{"time":"2016-02-16T16:31:25.352Z","level":"TRACE","msg":"Gossip Digests are : /10.0.2.117:1454676714:2922701
/10.0.2.168:1455284954:1077797 /10.0.1.129:1454674252:2930162 /10.0.2.128:1454668510:2947569
/10.0.1.32:1447859696:20667192 /10.0.1.12:1454675536:2926261 /10.0.1.88:1454668666:2947089
"}
{"time":"2016-02-16T16:31:25.354Z","level":"TRACE","msg":"Sending a GossipDigestSyn to /10.0.2.128
..."}
{"time":"2016-02-16T16:31:25.354Z","level":"TRACE","msg":"Performing status check ..."}
{"time":"2016-02-16T16:31:25.354Z","level":"TRACE","msg":"PHI for /10.0.2.128 : 0.14132872197630086"}
{"time":"2016-02-16T16:31:25.354Z","level":"DEBUG","msg":"time is expiring for endpoint :
/10.0.1.32 (1454932471326)"}
{"time":"2016-02-16T16:31:25.354Z","level":"DEBUG","msg":"evicting /10.0.1.32 from gossip"}
{"time":"2016-02-16T16:31:25.354Z","level":"TRACE","msg":"PHI for /10.0.1.129 : 0.7902186262859834"}
{"time":"2016-02-16T16:31:25.354Z","level":"TRACE","msg":"PHI for /10.0.2.168 : 1.2129498281352311"}
{"time":"2016-02-16T16:31:25.354Z","level":"TRACE","msg":"PHI for /10.0.1.12 : 1.1816270155595487"}
{"time":"2016-02-16T16:31:25.354Z","level":"TRACE","msg":"PHI for /10.0.2.117 : 0.13743310334362183"}
{"time":"2016-02-16T16:31:25.355Z","level":"TRACE","msg":"Received a GossipDigestAckMessage
from /10.0.2.128"}
{"time":"2016-02-16T16:31:25.355Z","level":"TRACE","msg":"Received ack with 1 digests and
1 states"}
{"time":"2016-02-16T16:31:25.355Z","level":"TRACE","msg":"/10.0.2.128local generation 1454668510,
remote generation 1454668510"}
{"time":"2016-02-16T16:31:25.355Z","level":"TRACE","msg":"Updating heartbeat state version
to 2947569 from 2947569 for /10.0.2.128 ..."}
{"time":"2016-02-16T16:31:25.355Z","level":"TRACE","msg":"local heartbeat version 2947089
greater than 2947088 for /10.0.1.88"}
{"time":"2016-02-16T16:31:25.355Z","level":"TRACE","msg":"Sending a GossipDigestAck2Message
to /10.0.2.128"}
{"time":"2016-02-16T16:31:25.542Z","level":"TRACE","msg":"local heartbeat version 2922701
greater than 2922698 for /10.0.2.117"}
{"time":"2016-02-16T16:31:25.542Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:25.542Z","level":"TRACE","msg":"local heartbeat version 2947089
greater than 2947086 for /10.0.1.88"}
{"time":"2016-02-16T16:31:25.542Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:25.542Z","level":"TRACE","msg":"local heartbeat version 2947569
greater than 2947566 for /10.0.2.128"}
{"time":"2016-02-16T16:31:25.542Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:25.543Z","level":"TRACE","msg":"Received a GossipDigestAck2Message
from /10.0.1.129"}
{"time":"2016-02-16T16:31:25.543Z","level":"TRACE","msg":"reporting /10.0.2.168"}
{"time":"2016-02-16T16:31:25.543Z","level":"TRACE","msg":"reporting /10.0.1.129"}
{"time":"2016-02-16T16:31:25.543Z","level":"TRACE","msg":"/10.0.2.168local generation 1455284954,
remote generation 1455284954"}
{"time":"2016-02-16T16:31:25.543Z","level":"TRACE","msg":"Updating heartbeat state version
to 1077801 from 1077795 for /10.0.2.168 ..."}
{"time":"2016-02-16T16:31:25.543Z","level":"TRACE","msg":"/10.0.1.129local generation 1454674252,
remote generation 1454674252"}
{"time":"2016-02-16T16:31:25.543Z","level":"TRACE","msg":"Updating heartbeat state version
to 2930165 from 2930162 for /10.0.1.129 ..."}
{"time":"2016-02-16T16:31:26.128Z","level":"TRACE","msg":"local heartbeat version 1077801
greater than 1077798 for /10.0.2.168"}
{"time":"2016-02-16T16:31:26.128Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.128Z","level":"TRACE","msg":"local heartbeat version 2922701
greater than 2922700 for /10.0.2.117"}
{"time":"2016-02-16T16:31:26.128Z","level":"TRACE","msg":"local heartbeat version 2930165
greater than 2930159 for /10.0.1.129"}
{"time":"2016-02-16T16:31:26.128Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.129Z","level":"TRACE","msg":"local heartbeat version 2947089
greater than 2947086 for /10.0.1.88"}
{"time":"2016-02-16T16:31:26.129Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.129Z","level":"TRACE","msg":"local heartbeat version 2947569
greater than 2947566 for /10.0.2.128"}
{"time":"2016-02-16T16:31:26.129Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.129Z","level":"TRACE","msg":"Received a GossipDigestAck2Message
from /10.0.1.12"}
{"time":"2016-02-16T16:31:26.129Z","level":"TRACE","msg":"reporting /10.0.1.12"}
{"time":"2016-02-16T16:31:26.129Z","level":"TRACE","msg":"/10.0.1.12local generation 1454675536,
remote generation 1454675536"}
{"time":"2016-02-16T16:31:26.129Z","level":"TRACE","msg":"Updating heartbeat state version
to 2926267 from 2926261 for /10.0.1.12 ..."}
{"time":"2016-02-16T16:31:26.130Z","level":"TRACE","msg":"local heartbeat version 1077801
greater than 1077798 for /10.0.2.168"}
{"time":"2016-02-16T16:31:26.130Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.130Z","level":"TRACE","msg":"local heartbeat version 2926267
greater than 2926264 for /10.0.1.12"}
{"time":"2016-02-16T16:31:26.130Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.130Z","level":"TRACE","msg":"local heartbeat version 2930165
greater than 2930159 for /10.0.1.129"}
{"time":"2016-02-16T16:31:26.130Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.130Z","level":"TRACE","msg":"local heartbeat version 2947089
greater than 2947086 for /10.0.1.88"}
{"time":"2016-02-16T16:31:26.130Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.130Z","level":"TRACE","msg":"local heartbeat version 2947569
greater than 2947568 for /10.0.2.128"}
{"time":"2016-02-16T16:31:26.130Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.131Z","level":"TRACE","msg":"Received a GossipDigestAck2Message
from /10.0.2.117"}
{"time":"2016-02-16T16:31:26.131Z","level":"TRACE","msg":"reporting /10.0.2.117"}
{"time":"2016-02-16T16:31:26.131Z","level":"TRACE","msg":"/10.0.2.117local generation 1454676714,
remote generation 1454676714"}
{"time":"2016-02-16T16:31:26.131Z","level":"TRACE","msg":"Updating heartbeat state version
to 2922704 from 2922701 for /10.0.2.117 ..."}
{"time":"2016-02-16T16:31:26.354Z","level":"TRACE","msg":"My heartbeat is now 2947092"}
{"time":"2016-02-16T16:31:26.354Z","level":"TRACE","msg":"Gossip Digests are : /10.0.2.168:1455284954:1077801
/10.0.1.88:1454668666:2947092 /10.0.2.128:1454668510:2947571 /10.0.1.129:1454674252:2930165
/10.0.1.12:1454675536:2926267 /10.0.2.117:1454676714:2922704 "}
{"time":"2016-02-16T16:31:26.354Z","level":"TRACE","msg":"Sending a GossipDigestSyn to /10.0.1.129
..."}
{"time":"2016-02-16T16:31:26.354Z","level":"TRACE","msg":"Performing status check ..."}
{"time":"2016-02-16T16:31:26.354Z","level":"TRACE","msg":"PHI for /10.0.2.128 : 1.1254232596265088"}
{"time":"2016-02-16T16:31:26.354Z","level":"TRACE","msg":"PHI for /10.0.1.129 : 0.7904647939808764"}
{"time":"2016-02-16T16:31:26.355Z","level":"TRACE","msg":"PHI for /10.0.2.168 : 0.803410322347404"}
{"time":"2016-02-16T16:31:26.355Z","level":"TRACE","msg":"PHI for /10.0.1.12 : 0.21743102018805321"}
{"time":"2016-02-16T16:31:26.355Z","level":"TRACE","msg":"PHI for /10.0.2.117 : 0.21360712465395978"}
{"time":"2016-02-16T16:31:26.355Z","level":"TRACE","msg":"Received a GossipDigestAckMessage
from /10.0.1.129"}
{"time":"2016-02-16T16:31:26.355Z","level":"TRACE","msg":"Received ack with 2 digests and
2 states"}
{"time":"2016-02-16T16:31:26.355Z","level":"TRACE","msg":"reporting /10.0.2.128"}
{"time":"2016-02-16T16:31:26.355Z","level":"TRACE","msg":"/10.0.2.128local generation 1454668510,
remote generation 1454668510"}
{"time":"2016-02-16T16:31:26.355Z","level":"TRACE","msg":"Updating heartbeat state version
to 2947572 from 2947569 for /10.0.2.128 ..."}
{"time":"2016-02-16T16:31:26.355Z","level":"TRACE","msg":"/10.0.1.129local generation 1454674252,
remote generation 1454674252"}
{"time":"2016-02-16T16:31:26.356Z","level":"TRACE","msg":"Updating heartbeat state version
to 2930165 from 2930165 for /10.0.1.129 ..."}
{"time":"2016-02-16T16:31:26.356Z","level":"TRACE","msg":"local heartbeat version 2926267
greater than 2926264 for /10.0.1.12"}
{"time":"2016-02-16T16:31:26.356Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.356Z","level":"TRACE","msg":"local heartbeat version 2947092
greater than 2947089 for /10.0.1.88"}
{"time":"2016-02-16T16:31:26.356Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.356Z","level":"TRACE","msg":"Sending a GossipDigestAck2Message
to /10.0.1.129"}
{"time":"2016-02-16T16:31:26.501Z","level":"TRACE","msg":"local heartbeat version 2922704
greater than 2922701 for /10.0.2.117"}
{"time":"2016-02-16T16:31:26.501Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.501Z","level":"TRACE","msg":"local heartbeat version 2930165
greater than 2930164 for /10.0.1.129"}
{"time":"2016-02-16T16:31:26.501Z","level":"TRACE","msg":"Adding state SEVERITY: 0.251889169216156"}
{"time":"2016-02-16T16:31:26.501Z","level":"TRACE","msg":"local heartbeat version 2947092
greater than 2947086 for /10.0.1.88"}
{"time":"2016-02-16T16:31:26.501Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.501Z","level":"TRACE","msg":"local heartbeat version 2947572
greater than 2947566 for /10.0.2.128"}
{"time":"2016-02-16T16:31:26.501Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.502Z","level":"TRACE","msg":"Received a GossipDigestAck2Message
from /10.0.2.168"}
{"time":"2016-02-16T16:31:26.502Z","level":"TRACE","msg":"reporting /10.0.2.168"}
{"time":"2016-02-16T16:31:26.502Z","level":"TRACE","msg":"/10.0.2.168local generation 1455284954,
remote generation 1455284954"}
{"time":"2016-02-16T16:31:26.502Z","level":"TRACE","msg":"Updating heartbeat state version
to 1077804 from 1077801 for /10.0.2.168 ..."}
{"time":"2016-02-16T16:31:27.130Z","level":"TRACE","msg":"local heartbeat version 1077804
greater than 1077801 for /10.0.2.168"}
{"time":"2016-02-16T16:31:27.130Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:27.130Z","level":"TRACE","msg":"local heartbeat version 2947092
greater than 2947091 for /10.0.1.88"}
{"time":"2016-02-16T16:31:27.130Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:27.130Z","level":"TRACE","msg":"local heartbeat version 2947572
greater than 2947571 for /10.0.2.128"}
{"time":"2016-02-16T16:31:27.131Z","level":"TRACE","msg":"Received a GossipDigestAck2Message
from /10.0.2.117"}
{"time":"2016-02-16T16:31:27.131Z","level":"TRACE","msg":"reporting /10.0.2.117"}
{"time":"2016-02-16T16:31:27.131Z","level":"TRACE","msg":"/10.0.2.117local generation 1454676714,
remote generation 1454676714"}
{"time":"2016-02-16T16:31:27.131Z","level":"TRACE","msg":"Updating heartbeat state version
to 2922707 from 2922704 for /10.0.2.117 ..."}
{"time":"2016-02-16T16:31:27.209Z","level":"TRACE","msg":"local heartbeat version 1077804
greater than 1077801 for /10.0.2.168"}
{"time":"2016-02-16T16:31:27.209Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:27.209Z","level":"TRACE","msg":"local heartbeat version 2922707
greater than 2922704 for /10.0.2.117"}
{"time":"2016-02-16T16:31:27.209Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:27.209Z","level":"TRACE","msg":"Adding state SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:27.209Z","level":"TRACE","msg":"requestAll for /10.0.1.32"}
{"time":"2016-02-16T16:31:27.211Z","level":"TRACE","msg":"Received a GossipDigestAck2Message
from /10.0.2.128"}
{"time":"2016-02-16T16:31:27.211Z","level":"TRACE","msg":"reporting /10.0.2.128"}
{"time":"2016-02-16T16:31:27.211Z","level":"TRACE","msg":"reporting /10.0.1.129"}
{"time":"2016-02-16T16:31:27.211Z","level":"TRACE","msg":"/10.0.2.128local generation 1454668510,
remote generation 1454668510"}
{"time":"2016-02-16T16:31:27.211Z","level":"TRACE","msg":"Updating heartbeat state version
to 2947575 from 2947572 for /10.0.2.128 ..."}
{"time":"2016-02-16T16:31:27.211Z","level":"TRACE","msg":"Ignoring gossip for /10.0.1.32 because
it is quarantined"}
{noformat}

> Decommissioned nodes can remain in gossip
> -----------------------------------------
>
>                 Key: CASSANDRA-10371
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10371
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Distributed Metadata
>            Reporter: Brandon Williams
>            Assignee: Stefania
>            Priority: Minor
>
> This may apply to other dead states as well.  Dead states should be expired after 3 days.
 In the case of decom we attach a timestamp to let the other nodes know when it should be
expired.  It has been observed that sometimes a subset of nodes in the cluster never expire
the state, and through heap analysis of these nodes it is revealed that the epstate.isAlive
check returns true when it should return false, which would allow the state to be evicted.
 This may have been affected by CASSANDRA-8336.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message