cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "pavel (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-10371) Decommissioned nodes can remain in gossip
Date Thu, 18 Feb 2016 18:07:19 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15152743#comment-15152743
] 

pavel edited comment on CASSANDRA-10371 at 2/18/16 6:07 PM:
------------------------------------------------------------

And log from the time of decommission. It is all info about 10.0.1.32
{noformat}
at  system.log.2016-02-12_1126 | grep 10.0.1.32
{"time":"2016-02-05T10:35:10.066Z","level":"INFO","msg":"Node /10.0.1.32 is now part of the
cluster"}
{"time":"2016-02-05T10:35:10.099Z","level":"INFO","msg":"Handshaking version with /10.0.1.32"}
{"time":"2016-02-05T10:35:10.220Z","level":"INFO","msg":"InetAddress /10.0.1.32 is now UP"}
{"time":"2016-02-05T10:35:12.563Z","level":"INFO","msg":"Node /10.0.1.32 is now part of the
cluster"}
{"time":"2016-02-05T10:35:12.565Z","level":"INFO","msg":"InetAddress /10.0.1.32 is now UP"}
{"time":"2016-02-05T10:35:12.607Z","level":"INFO","msg":"Updating topology for /10.0.1.32"}
{"time":"2016-02-05T10:35:12.608Z","level":"INFO","msg":"Updating topology for /10.0.1.32"}
{"time":"2016-02-05T10:35:12.610Z","level":"INFO","msg":"Handshaking version with /10.0.1.32"}
{"time":"2016-02-05T10:35:47.461Z","level":"INFO","msg":"[Stream #369150e0-cbf4-11e5-9a6b-fd238f43a3db]
Starting streaming to /10.0.1.32"}
{"time":"2016-02-05T10:35:47.557Z","level":"INFO","msg":"[Stream #369150e0-cbf4-11e5-9a6b-fd238f43a3db,
ID#0] Beginning stream session with /10.0.1.32"}
{"time":"2016-02-05T10:37:04.863Z","level":"INFO","msg":"[Stream #369150e0-cbf4-11e5-9a6b-fd238f43a3db]
Session with /10.0.1.32 is complete"}
{"time":"2016-02-05T11:54:27.374Z","level":"INFO","msg":"[Stream #177c4fb0-cbff-11e5-8ce1-a53fdfde2603]
Session with /10.0.1.32 is complete"}
{"time":"2016-02-05T11:54:32.129Z","level":"INFO","msg":"Removing tokens [-1183123795032699037,
-1237172198872744999, -1270149265263593564, -1389400073696907418, -1402231454860468978, -1420312602012480180,
-1627335600868165275, -1713896876933182948, -1714549883939918780, -1741474774932965377, -1792210053935902256,
-1807601543024917385, -1854566658540932161, -1912378675308644559, -1963516450452847956, -2081077201955154820,
-2099259848054237265, -2129680696766320789, -2221883494221453885, -2310616694711668360, -2331548641412448139,
-2334622440252028474, -2411111763982847025, -245865888987490824, -246385138307167507, -2473543956019538623,
-2490169864633659751, -2532042510174406079, -2540038371895843059, -2577021784711590342, -2593326939593610701,
-2705600331335690795, -2770464252902532656, -2839608713233822191, -2912475620975565625, -2943366603912110567,
-2949295810903492735, -2954629576212069889, -3095752208331422097, -3121066151673234082, -3185818042633507184,
-3219579857569752331, -3286343234413829618, -3289689335761352230, -3361565551667832402, -3383031027800371008,
-3412816001810159961, -3457683958466098497, -3508695836421455440, -3532532777067013075, -3543699029518234044,
-3580715661159129955, -3599252547477456660, -3661459590777641370, -3667538041993955650, -3690186824731493069,
-3814115795769795724, -409080988277749690, -4156007243077529438, -4168774893038102265, -4213854018229603756,
-4249034660629583574, -4281296366845033280, -4320926472067799077, -4545355489460541553, -4630628418422675470,
-4654983718222492637, -4768547837932983920, -5159507483120816494, -51965338274886423, -5199723923113410828,
-5223817829501441469, -5267064983314818217, -5348036139422617554, -5364643938379720849, -5495473223858010026,
-5538859875930354329, -554259440667407064, -5547533646691145209, -5599903631185515707, -5779719603131964183,
-5932457424237549800, -6205324647087801173, -6229187725024886147, -6231270541967013773, -6235596621005793282,
-6243196665962158041, -6264073286569207959, -6427783773364474000, -6501124205004483449, -6515615874324067226,
-6573166604041811188, -6637220677879215528, -6651782973192407164, -6660806062319128258, -6745026564913028874,
-6812331251174808386, -7084494716757599690, -7149011211591734620, -7274798955118474541, -731652465373161574,
-7403663263352230504, -7447942789148217722, -7449824614964680985, -7587057746419387993, -7641766861615411515,
-7685202236673643074, -7707429459189222115, -7742454891260223055, -7833962528790251697, -7931725553821464164,
-7939773289897241357, -8098012638292660116, -8219265643204532541, -8235265212011847570, -8252620209307958750,
-8332424581430025253, -8349765834148048802, -836487426771203231, -8505914076204635715, -851663313261657661,
-8519826107865925909, -8695372492829159498, -8942575831111517062, -8976797677584625277, -9051099539809741903,
-963992846195142625, 1003830146651642498, 1178027849963644975, 1224976256801927215, 1244352078631038965,
1417164721674217220, 1451341591539912399, 1491082438042287951, 1514848536642096395, 1779901662209221643,
1854090933062166699, 1943140502461231414, 1994380979939543148, 2007534562233247106, 217709005954998613,
2235989942794609624, 2241300505721899121, 2246701380267263353, 231183097291070161, 2426913079401922388,
2451699806267654588, 2506037868809622944, 2539177368085335242, 2585691867824138041, 2615916489656464006,
2629622650576085045, 2657709892891143999, 269267015073834956, 2978608649954009655, 2994749269564046800,
3002378570488766614, 3121394566209939036, 3141180417712080576, 3154166401188000671, 3295643071378191280,
3367253256599281972, 3373165743103532333, 3410319489097671220, 341173714064808701, 35319237546365581,
358168701228763255, 3695798865479120909, 3784007090181772945, 3859215587032554422, 388414320421460408,
3958884439975359895, 4116791292047135794, 4157923384507172767, 4177479403175856644, 4192843956567907187,
4237835213950914050, 4283455088657318777, 4437755738583404917, 4464954521637384793, 4479350931740982844,
448168435071015674, 4483893029099418629, 4495279466361191724, 4626894508809678069, 4913067912592229158,
5055224510936602124, 5146899889661568824, 5333227006937586923, 5344713819757688419, 5354664536118782383,
5407856766112305920, 5524147618224097934, 5530799914723849578, 5594217902784391863, 5740424123774088150,
5773813700926694122, 5797583289244413653, 596003237801815323, 6051342740892152639, 6053457919557951562,
6063733117952809262, 6188792405330710479, 6215325692940778409, 6255014464234787541, 6307265543356112469,
6331411861541807584, 6428394292228979654, 6440759262443849168, 6519598944774579329, 6559591672744555181,
6637797466881599715, 6662404025200321225, 6668777379548711499, 6828109123001551836, 6861072227483193626,
6876892389046410599, 6982428538201939129, 7004630263067699041, 703529510861716420, 7035478043169974728,
713068522342900631, 7143695289303246630, 7149047549904215441, 7302396691607427712, 7315761441890909840,
7410347447605570455, 754235745869667971, 7544573809574793144, 7586097788690287546, 7597038064500920869,
7614889355576709597, 7624148922826428440, 7627724472890172915, 7896536432019982357, 8070449930340402428,
8100741102346464407, 8206476157175887120, 8277001814877694995, 8323125797400536081, 8331290716255307824,
8338939509697372178, 834182466138639995, 8388253326997854443, 8414277759665359090, 8670923111306643765,
8769994749009616915, 8770706585661243637, 8812518846508548955, 8847966629758817746, 8906298870427014138,
8987438727642595319, 9050092026849126476, 933979495781858352, 962613924858074837, 985754492835172659]
for /10.0.1.32"}
{"time":"2016-02-05T11:54:32.138Z","level":"INFO","msg":"Deleting any stored hints for /10.0.1.32"}
{"time":"2016-02-05T11:54:32.161Z","level":"INFO","msg":"Handshaking version with /10.0.1.32"}
{noformat}


was (Author: pontryagin):
And log from the time of decommission
{noformat}
at  system.log.2016-02-12_1126 | grep 10.0.1.32
{"time":"2016-02-05T10:35:10.066Z","level":"INFO","msg":"Node /10.0.1.32 is now part of the
cluster"}
{"time":"2016-02-05T10:35:10.099Z","level":"INFO","msg":"Handshaking version with /10.0.1.32"}
{"time":"2016-02-05T10:35:10.220Z","level":"INFO","msg":"InetAddress /10.0.1.32 is now UP"}
{"time":"2016-02-05T10:35:12.563Z","level":"INFO","msg":"Node /10.0.1.32 is now part of the
cluster"}
{"time":"2016-02-05T10:35:12.565Z","level":"INFO","msg":"InetAddress /10.0.1.32 is now UP"}
{"time":"2016-02-05T10:35:12.607Z","level":"INFO","msg":"Updating topology for /10.0.1.32"}
{"time":"2016-02-05T10:35:12.608Z","level":"INFO","msg":"Updating topology for /10.0.1.32"}
{"time":"2016-02-05T10:35:12.610Z","level":"INFO","msg":"Handshaking version with /10.0.1.32"}
{"time":"2016-02-05T10:35:47.461Z","level":"INFO","msg":"[Stream #369150e0-cbf4-11e5-9a6b-fd238f43a3db]
Starting streaming to /10.0.1.32"}
{"time":"2016-02-05T10:35:47.557Z","level":"INFO","msg":"[Stream #369150e0-cbf4-11e5-9a6b-fd238f43a3db,
ID#0] Beginning stream session with /10.0.1.32"}
{"time":"2016-02-05T10:37:04.863Z","level":"INFO","msg":"[Stream #369150e0-cbf4-11e5-9a6b-fd238f43a3db]
Session with /10.0.1.32 is complete"}
{"time":"2016-02-05T11:54:27.374Z","level":"INFO","msg":"[Stream #177c4fb0-cbff-11e5-8ce1-a53fdfde2603]
Session with /10.0.1.32 is complete"}
{"time":"2016-02-05T11:54:32.129Z","level":"INFO","msg":"Removing tokens [-1183123795032699037,
-1237172198872744999, -1270149265263593564, -1389400073696907418, -1402231454860468978, -1420312602012480180,
-1627335600868165275, -1713896876933182948, -1714549883939918780, -1741474774932965377, -1792210053935902256,
-1807601543024917385, -1854566658540932161, -1912378675308644559, -1963516450452847956, -2081077201955154820,
-2099259848054237265, -2129680696766320789, -2221883494221453885, -2310616694711668360, -2331548641412448139,
-2334622440252028474, -2411111763982847025, -245865888987490824, -246385138307167507, -2473543956019538623,
-2490169864633659751, -2532042510174406079, -2540038371895843059, -2577021784711590342, -2593326939593610701,
-2705600331335690795, -2770464252902532656, -2839608713233822191, -2912475620975565625, -2943366603912110567,
-2949295810903492735, -2954629576212069889, -3095752208331422097, -3121066151673234082, -3185818042633507184,
-3219579857569752331, -3286343234413829618, -3289689335761352230, -3361565551667832402, -3383031027800371008,
-3412816001810159961, -3457683958466098497, -3508695836421455440, -3532532777067013075, -3543699029518234044,
-3580715661159129955, -3599252547477456660, -3661459590777641370, -3667538041993955650, -3690186824731493069,
-3814115795769795724, -409080988277749690, -4156007243077529438, -4168774893038102265, -4213854018229603756,
-4249034660629583574, -4281296366845033280, -4320926472067799077, -4545355489460541553, -4630628418422675470,
-4654983718222492637, -4768547837932983920, -5159507483120816494, -51965338274886423, -5199723923113410828,
-5223817829501441469, -5267064983314818217, -5348036139422617554, -5364643938379720849, -5495473223858010026,
-5538859875930354329, -554259440667407064, -5547533646691145209, -5599903631185515707, -5779719603131964183,
-5932457424237549800, -6205324647087801173, -6229187725024886147, -6231270541967013773, -6235596621005793282,
-6243196665962158041, -6264073286569207959, -6427783773364474000, -6501124205004483449, -6515615874324067226,
-6573166604041811188, -6637220677879215528, -6651782973192407164, -6660806062319128258, -6745026564913028874,
-6812331251174808386, -7084494716757599690, -7149011211591734620, -7274798955118474541, -731652465373161574,
-7403663263352230504, -7447942789148217722, -7449824614964680985, -7587057746419387993, -7641766861615411515,
-7685202236673643074, -7707429459189222115, -7742454891260223055, -7833962528790251697, -7931725553821464164,
-7939773289897241357, -8098012638292660116, -8219265643204532541, -8235265212011847570, -8252620209307958750,
-8332424581430025253, -8349765834148048802, -836487426771203231, -8505914076204635715, -851663313261657661,
-8519826107865925909, -8695372492829159498, -8942575831111517062, -8976797677584625277, -9051099539809741903,
-963992846195142625, 1003830146651642498, 1178027849963644975, 1224976256801927215, 1244352078631038965,
1417164721674217220, 1451341591539912399, 1491082438042287951, 1514848536642096395, 1779901662209221643,
1854090933062166699, 1943140502461231414, 1994380979939543148, 2007534562233247106, 217709005954998613,
2235989942794609624, 2241300505721899121, 2246701380267263353, 231183097291070161, 2426913079401922388,
2451699806267654588, 2506037868809622944, 2539177368085335242, 2585691867824138041, 2615916489656464006,
2629622650576085045, 2657709892891143999, 269267015073834956, 2978608649954009655, 2994749269564046800,
3002378570488766614, 3121394566209939036, 3141180417712080576, 3154166401188000671, 3295643071378191280,
3367253256599281972, 3373165743103532333, 3410319489097671220, 341173714064808701, 35319237546365581,
358168701228763255, 3695798865479120909, 3784007090181772945, 3859215587032554422, 388414320421460408,
3958884439975359895, 4116791292047135794, 4157923384507172767, 4177479403175856644, 4192843956567907187,
4237835213950914050, 4283455088657318777, 4437755738583404917, 4464954521637384793, 4479350931740982844,
448168435071015674, 4483893029099418629, 4495279466361191724, 4626894508809678069, 4913067912592229158,
5055224510936602124, 5146899889661568824, 5333227006937586923, 5344713819757688419, 5354664536118782383,
5407856766112305920, 5524147618224097934, 5530799914723849578, 5594217902784391863, 5740424123774088150,
5773813700926694122, 5797583289244413653, 596003237801815323, 6051342740892152639, 6053457919557951562,
6063733117952809262, 6188792405330710479, 6215325692940778409, 6255014464234787541, 6307265543356112469,
6331411861541807584, 6428394292228979654, 6440759262443849168, 6519598944774579329, 6559591672744555181,
6637797466881599715, 6662404025200321225, 6668777379548711499, 6828109123001551836, 6861072227483193626,
6876892389046410599, 6982428538201939129, 7004630263067699041, 703529510861716420, 7035478043169974728,
713068522342900631, 7143695289303246630, 7149047549904215441, 7302396691607427712, 7315761441890909840,
7410347447605570455, 754235745869667971, 7544573809574793144, 7586097788690287546, 7597038064500920869,
7614889355576709597, 7624148922826428440, 7627724472890172915, 7896536432019982357, 8070449930340402428,
8100741102346464407, 8206476157175887120, 8277001814877694995, 8323125797400536081, 8331290716255307824,
8338939509697372178, 834182466138639995, 8388253326997854443, 8414277759665359090, 8670923111306643765,
8769994749009616915, 8770706585661243637, 8812518846508548955, 8847966629758817746, 8906298870427014138,
8987438727642595319, 9050092026849126476, 933979495781858352, 962613924858074837, 985754492835172659]
for /10.0.1.32"}
{"time":"2016-02-05T11:54:32.138Z","level":"INFO","msg":"Deleting any stored hints for /10.0.1.32"}
{"time":"2016-02-05T11:54:32.161Z","level":"INFO","msg":"Handshaking version with /10.0.1.32"}
{noformat}

> Decommissioned nodes can remain in gossip
> -----------------------------------------
>
>                 Key: CASSANDRA-10371
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10371
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Distributed Metadata
>            Reporter: Brandon Williams
>            Assignee: Joel Knighton
>            Priority: Minor
>
> This may apply to other dead states as well.  Dead states should be expired after 3 days.
 In the case of decom we attach a timestamp to let the other nodes know when it should be
expired.  It has been observed that sometimes a subset of nodes in the cluster never expire
the state, and through heap analysis of these nodes it is revealed that the epstate.isAlive
check returns true when it should return false, which would allow the state to be evicted.
 This may have been affected by CASSANDRA-8336.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message