cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "pavel (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-10371) Decommissioned nodes can remain in gossip
Date Mon, 15 Feb 2016 10:29:18 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15147176#comment-15147176
] 

pavel edited comment on CASSANDRA-10371 at 2/15/16 10:29 AM:
-------------------------------------------------------------

Attached the logs. I will be able to restart in debug mode later. Hopefully it a couple of
weeks.

{noformat}
{"time":"2016-02-15T10:17:27.846Z","level":"INFO","msg":"InetAddress /10.0.1.32 is now DOWN"}
{"time":"2016-02-15T10:17:27.847Z","level":"INFO","msg":"Removing tokens [-1183123795032699037,
  -1237172198872744999, -1270149265263593564, -1389400073696907418, -1402231454860468978,
-1420312602012480180, -1627335600868165275, -1713896876933182948, -1714549883939918780, -1741474774932965377,
-1792210053935902256, -1807601543024917385, -1854566658540932161, -1912378675308644559, -1963516450452847956,
-2081077201955154820, -2099259848054237265, -2129680696766320789, -2221883494221453885, -2310616694711668360,
-2331548641412448139, -2334622440252028474, -2411111763982847025, -245865888987490824, -246385138307167507,
-2473543956019538623, -2490169864633659751, -2532042510174406079, -2540038371895843059, -2577021784711590342,
-2593326939593610701, -2705600331335690795, -2770464252902532656, -2839608713233822191, -2912475620975565625,
-2943366603912110567, -2949295810903492735, -2954629576212069889, -3095752208331422097, -3121066151673234082,
-3185818042633507184, -3219579857569752331, -3286343234413829618, -3289689335761352230, -3361565551667832402,
-3383031027800371008, -3412816001810159961, -3457683958466098497, -3508695836421455440, -3532532777067013075,
-3543699029518234044, -3580715661159129955, -3599252547477456660, -3661459590777641370, -3667538041993955650,
-3690186824731493069, -3814115795769795724, -409080988277749690, -4156007243077529438, -4168774893038102265,
-4213854018229603756, -4249034660629583574, -4281296366845033280, -4320926472067799077, -4545355489460541553,
-4630628418422675470, -4654983718222492637, -4768547837932983920, -5159507483120816494, -51965338274886423,
-5199723923113410828, -5223817829501441469, -5267064983314818217, -5348036139422617554, -5364643938379720849,
-5495473223858010026, -5538859875930354329, -554259440667407064, -5547533646691145209, -5599903631185515707,
-5779719603131964183, -5932457424237549800, -6205324647087801173, -6229187725024886147, -6231270541967013773,
-6235596621005793282, -6243196665962158041, -6264073286569207959, -6427783773364474000, -6501124205004483449,
-6515615874324067226, -6573166604041811188, -6637220677879215528, -6651782973192407164, -6660806062319128258,
-6745026564913028874, -6812331251174808386, -7084494716757599690, -7149011211591734620, -7274798955118474541,
-731652465373161574, -7403663263352230504, -7447942789148217722, -7449824614964680985, -7587057746419387993,
-7641766861615411515, -7685202236673643074, -7707429459189222115, -7742454891260223055, -7833962528790251697,
-7931725553821464164, -7939773289897241357, -8098012638292660116, -8219265643204532541, -8235265212011847570,
-8252620209307958750, -8332424581430025253, -8349765834148048802, -836487426771203231, -8505914076204635715,
-851663313261657661, -8519826107865925909, -8695372492829159498, -8942575831111517062, -8976797677584625277,
-9051099539809741903, -963992846195142625, 1003830146651642498, 1178027849963644975, 1224976256801927215,
1244352078631038965, 1417164721674217220, 1451341591539912399, 1491082438042287951, 1514848536642096395,
1779901662209221643, 1854090933062166699, 1943140502461231414, 1994380979939543148, 2007534562233247106,
217709005954998613, 2235989942794609624, 2241300505721899121, 2246701380267263353, 231183097291070161,
2426913079401922388, 2451699806267654588, 2506037868809622944, 2539177368085335242, 2585691867824138041,
2615916489656464006, 2629622650576085045, 2657709892891143999, 269267015073834956, 2978608649954009655,
2994749269564046800, 3002378570488766614, 3121394566209939036, 3141180417712080576, 3154166401188000671,
3295643071378191280, 3367253256599281972, 3373165743103532333, 3410319489097671220, 341173714064808701,
35319237546365581, 358168701228763255, 3695798865479120909, 3784007090181772945, 3859215587032554422,
388414320421460408, 3958884439975359895, 4116791292047135794, 4157923384507172767, 4177479403175856644,
4192843956567907187, 4237835213950914050, 4283455088657318777, 4437755738583404917, 4464954521637384793,
4479350931740982844, 448168435071015674, 4483893029099418629, 4495279466361191724, 4626894508809678069,
4913067912592229158, 5055224510936602124, 5146899889661568824, 5333227006937586923, 5344713819757688419,
5354664536118782383, 5407856766112305920, 5524147618224097934, 5530799914723849578, 5594217902784391863,
5740424123774088150, 5773813700926694122, 5797583289244413653, 596003237801815323, 6051342740892152639,
6053457919557951562, 6063733117952809262, 6188792405330710479, 6215325692940778409, 6255014464234787541,
6307265543356112469, 6331411861541807584, 6428394292228979654, 6440759262443849168, 6519598944774579329,
6559591672744555181, 6637797466881599715, 6662404025200321225, 6668777379548711499, 6828109123001551836,
6861072227483193626, 6876892389046410599, 6982428538201939129, 7004630263067699041, 703529510861716420,
7035478043169974728, 713068522342900631, 7143695289303246630, 7149047549904215441, 7302396691607427712,
7315761441890909840, 7410347447605570455, 754235745869667971, 7544573809574793144, 7586097788690287546,
7597038064500920869, 7614889355576709597, 7624148922826428440, 7627724472890172915, 7896536432019982357,
8070449930340402428, 8100741102346464407, 8206476157175887120, 8277001814877694995, 8323125797400536081,
8331290716255307824, 8338939509697372178, 834182466138639995, 8388253326997854443, 8414277759665359090,
8670923111306643765, 8769994749009616915, 8770706585661243637, 8812518846508548955, 8847966629758817746,
8906298870427014138, 8987438727642595319, 9050092026849126476, 933979495781858352, 962613924858074837,
985754492835172659] for /10.0.1.32"}
{"time":"2016-02-15T10:18:36.860Z","level":"INFO","msg":"InetAddress /10.0.1.32 is now DOWN"}
{noformat}


was (Author: pontryagin):
Attached the logs. I will be able to restart in debug mode later. Hopefully it a couple of
weeks.

 {"time":"2016-02-15T10:17:27.846Z","level":"INFO","msg":"InetAddress /10.0.1.32 is now DOWN"}
 {"time":"2016-02-15T10:17:27.847Z","level":"INFO","msg":"Removing tokens [-1183123795032699037,
  -1237172198872744999, -1270149265263593564, -1389400073696907418, -1402231454860468978,
-1420312602012480180, -1627335600868165275, -1713896876933182948, -1714549883939918780, -1741474774932965377,
-1792210053935902256, -1807601543024917385, -1854566658540932161, -1912378675308644559, -1963516450452847956,
-2081077201955154820, -2099259848054237265, -2129680696766320789, -2221883494221453885, -2310616694711668360,
-2331548641412448139, -2334622440252028474, -2411111763982847025, -245865888987490824, -246385138307167507,
-2473543956019538623, -2490169864633659751, -2532042510174406079, -2540038371895843059, -2577021784711590342,
-2593326939593610701, -2705600331335690795, -2770464252902532656, -2839608713233822191, -2912475620975565625,
-2943366603912110567, -2949295810903492735, -2954629576212069889, -3095752208331422097, -3121066151673234082,
-3185818042633507184, -3219579857569752331, -3286343234413829618, -3289689335761352230, -3361565551667832402,
-3383031027800371008, -3412816001810159961, -3457683958466098497, -3508695836421455440, -3532532777067013075,
-3543699029518234044, -3580715661159129955, -3599252547477456660, -3661459590777641370, -3667538041993955650,
-3690186824731493069, -3814115795769795724, -409080988277749690, -4156007243077529438, -4168774893038102265,
-4213854018229603756, -4249034660629583574, -4281296366845033280, -4320926472067799077, -4545355489460541553,
-4630628418422675470, -4654983718222492637, -4768547837932983920, -5159507483120816494, -51965338274886423,
-5199723923113410828, -5223817829501441469, -5267064983314818217, -5348036139422617554, -5364643938379720849,
-5495473223858010026, -5538859875930354329, -554259440667407064, -5547533646691145209, -5599903631185515707,
-5779719603131964183, -5932457424237549800, -6205324647087801173, -6229187725024886147, -6231270541967013773,
-6235596621005793282, -6243196665962158041, -6264073286569207959, -6427783773364474000, -6501124205004483449,
-6515615874324067226, -6573166604041811188, -6637220677879215528, -6651782973192407164, -6660806062319128258,
-6745026564913028874, -6812331251174808386, -7084494716757599690, -7149011211591734620, -7274798955118474541,
-731652465373161574, -7403663263352230504, -7447942789148217722, -7449824614964680985, -7587057746419387993,
-7641766861615411515, -7685202236673643074, -7707429459189222115, -7742454891260223055, -7833962528790251697,
-7931725553821464164, -7939773289897241357, -8098012638292660116, -8219265643204532541, -8235265212011847570,
-8252620209307958750, -8332424581430025253, -8349765834148048802, -836487426771203231, -8505914076204635715,
-851663313261657661, -8519826107865925909, -8695372492829159498, -8942575831111517062, -8976797677584625277,
-9051099539809741903, -963992846195142625, 1003830146651642498, 1178027849963644975, 1224976256801927215,
1244352078631038965, 1417164721674217220, 1451341591539912399, 1491082438042287951, 1514848536642096395,
1779901662209221643, 1854090933062166699, 1943140502461231414, 1994380979939543148, 2007534562233247106,
217709005954998613, 2235989942794609624, 2241300505721899121, 2246701380267263353, 231183097291070161,
2426913079401922388, 2451699806267654588, 2506037868809622944, 2539177368085335242, 2585691867824138041,
2615916489656464006, 2629622650576085045, 2657709892891143999, 269267015073834956, 2978608649954009655,
2994749269564046800, 3002378570488766614, 3121394566209939036, 3141180417712080576, 3154166401188000671,
3295643071378191280, 3367253256599281972, 3373165743103532333, 3410319489097671220, 341173714064808701,
35319237546365581, 358168701228763255, 3695798865479120909, 3784007090181772945, 3859215587032554422,
388414320421460408, 3958884439975359895, 4116791292047135794, 4157923384507172767, 4177479403175856644,
4192843956567907187, 4237835213950914050, 4283455088657318777, 4437755738583404917, 4464954521637384793,
4479350931740982844, 448168435071015674, 4483893029099418629, 4495279466361191724, 4626894508809678069,
4913067912592229158, 5055224510936602124, 5146899889661568824, 5333227006937586923, 5344713819757688419,
5354664536118782383, 5407856766112305920, 5524147618224097934, 5530799914723849578, 5594217902784391863,
5740424123774088150, 5773813700926694122, 5797583289244413653, 596003237801815323, 6051342740892152639,
6053457919557951562, 6063733117952809262, 6188792405330710479, 6215325692940778409, 6255014464234787541,
6307265543356112469, 6331411861541807584, 6428394292228979654, 6440759262443849168, 6519598944774579329,
6559591672744555181, 6637797466881599715, 6662404025200321225, 6668777379548711499, 6828109123001551836,
6861072227483193626, 6876892389046410599, 6982428538201939129, 7004630263067699041, 703529510861716420,
7035478043169974728, 713068522342900631, 7143695289303246630, 7149047549904215441, 7302396691607427712,
7315761441890909840, 7410347447605570455, 754235745869667971, 7544573809574793144, 7586097788690287546,
7597038064500920869, 7614889355576709597, 7624148922826428440, 7627724472890172915, 7896536432019982357,
8070449930340402428, 8100741102346464407, 8206476157175887120, 8277001814877694995, 8323125797400536081,
8331290716255307824, 8338939509697372178, 834182466138639995, 8388253326997854443, 8414277759665359090,
8670923111306643765, 8769994749009616915, 8770706585661243637, 8812518846508548955, 8847966629758817746,
8906298870427014138, 8987438727642595319, 9050092026849126476, 933979495781858352, 962613924858074837,
985754492835172659] for /10.0.1.32"}
 {"time":"2016-02-15T10:18:36.860Z","level":"INFO","msg":"InetAddress /10.0.1.32 is now DOWN"}


> Decommissioned nodes can remain in gossip
> -----------------------------------------
>
>                 Key: CASSANDRA-10371
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10371
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Distributed Metadata
>            Reporter: Brandon Williams
>            Assignee: Stefania
>            Priority: Minor
>
> This may apply to other dead states as well.  Dead states should be expired after 3 days.
 In the case of decom we attach a timestamp to let the other nodes know when it should be
expired.  It has been observed that sometimes a subset of nodes in the cluster never expire
the state, and through heap analysis of these nodes it is revealed that the epstate.isAlive
check returns true when it should return false, which would allow the state to be evicted.
 This may have been affected by CASSANDRA-8336.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message