cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "pavel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10371) Decommissioned nodes can remain in gossip
Date Thu, 18 Feb 2016 18:07:18 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15152743#comment-15152743
] 

pavel commented on CASSANDRA-10371:
-----------------------------------

And log from the time of decommission
{noformat}
at  system.log.2016-02-12_1126 | grep 10.0.1.32
{"time":"2016-02-05T10:35:10.066Z","level":"INFO","msg":"Node /10.0.1.32 is now part of the
cluster"}
{"time":"2016-02-05T10:35:10.099Z","level":"INFO","msg":"Handshaking version with /10.0.1.32"}
{"time":"2016-02-05T10:35:10.220Z","level":"INFO","msg":"InetAddress /10.0.1.32 is now UP"}
{"time":"2016-02-05T10:35:12.563Z","level":"INFO","msg":"Node /10.0.1.32 is now part of the
cluster"}
{"time":"2016-02-05T10:35:12.565Z","level":"INFO","msg":"InetAddress /10.0.1.32 is now UP"}
{"time":"2016-02-05T10:35:12.607Z","level":"INFO","msg":"Updating topology for /10.0.1.32"}
{"time":"2016-02-05T10:35:12.608Z","level":"INFO","msg":"Updating topology for /10.0.1.32"}
{"time":"2016-02-05T10:35:12.610Z","level":"INFO","msg":"Handshaking version with /10.0.1.32"}
{"time":"2016-02-05T10:35:47.461Z","level":"INFO","msg":"[Stream #369150e0-cbf4-11e5-9a6b-fd238f43a3db]
Starting streaming to /10.0.1.32"}
{"time":"2016-02-05T10:35:47.557Z","level":"INFO","msg":"[Stream #369150e0-cbf4-11e5-9a6b-fd238f43a3db,
ID#0] Beginning stream session with /10.0.1.32"}
{"time":"2016-02-05T10:37:04.863Z","level":"INFO","msg":"[Stream #369150e0-cbf4-11e5-9a6b-fd238f43a3db]
Session with /10.0.1.32 is complete"}
{"time":"2016-02-05T11:54:27.374Z","level":"INFO","msg":"[Stream #177c4fb0-cbff-11e5-8ce1-a53fdfde2603]
Session with /10.0.1.32 is complete"}
{"time":"2016-02-05T11:54:32.129Z","level":"INFO","msg":"Removing tokens [-1183123795032699037,
-1237172198872744999, -1270149265263593564, -1389400073696907418, -1402231454860468978, -1420312602012480180,
-1627335600868165275, -1713896876933182948, -1714549883939918780, -1741474774932965377, -1792210053935902256,
-1807601543024917385, -1854566658540932161, -1912378675308644559, -1963516450452847956, -2081077201955154820,
-2099259848054237265, -2129680696766320789, -2221883494221453885, -2310616694711668360, -2331548641412448139,
-2334622440252028474, -2411111763982847025, -245865888987490824, -246385138307167507, -2473543956019538623,
-2490169864633659751, -2532042510174406079, -2540038371895843059, -2577021784711590342, -2593326939593610701,
-2705600331335690795, -2770464252902532656, -2839608713233822191, -2912475620975565625, -2943366603912110567,
-2949295810903492735, -2954629576212069889, -3095752208331422097, -3121066151673234082, -3185818042633507184,
-3219579857569752331, -3286343234413829618, -3289689335761352230, -3361565551667832402, -3383031027800371008,
-3412816001810159961, -3457683958466098497, -3508695836421455440, -3532532777067013075, -3543699029518234044,
-3580715661159129955, -3599252547477456660, -3661459590777641370, -3667538041993955650, -3690186824731493069,
-3814115795769795724, -409080988277749690, -4156007243077529438, -4168774893038102265, -4213854018229603756,
-4249034660629583574, -4281296366845033280, -4320926472067799077, -4545355489460541553, -4630628418422675470,
-4654983718222492637, -4768547837932983920, -5159507483120816494, -51965338274886423, -5199723923113410828,
-5223817829501441469, -5267064983314818217, -5348036139422617554, -5364643938379720849, -5495473223858010026,
-5538859875930354329, -554259440667407064, -5547533646691145209, -5599903631185515707, -5779719603131964183,
-5932457424237549800, -6205324647087801173, -6229187725024886147, -6231270541967013773, -6235596621005793282,
-6243196665962158041, -6264073286569207959, -6427783773364474000, -6501124205004483449, -6515615874324067226,
-6573166604041811188, -6637220677879215528, -6651782973192407164, -6660806062319128258, -6745026564913028874,
-6812331251174808386, -7084494716757599690, -7149011211591734620, -7274798955118474541, -731652465373161574,
-7403663263352230504, -7447942789148217722, -7449824614964680985, -7587057746419387993, -7641766861615411515,
-7685202236673643074, -7707429459189222115, -7742454891260223055, -7833962528790251697, -7931725553821464164,
-7939773289897241357, -8098012638292660116, -8219265643204532541, -8235265212011847570, -8252620209307958750,
-8332424581430025253, -8349765834148048802, -836487426771203231, -8505914076204635715, -851663313261657661,
-8519826107865925909, -8695372492829159498, -8942575831111517062, -8976797677584625277, -9051099539809741903,
-963992846195142625, 1003830146651642498, 1178027849963644975, 1224976256801927215, 1244352078631038965,
1417164721674217220, 1451341591539912399, 1491082438042287951, 1514848536642096395, 1779901662209221643,
1854090933062166699, 1943140502461231414, 1994380979939543148, 2007534562233247106, 217709005954998613,
2235989942794609624, 2241300505721899121, 2246701380267263353, 231183097291070161, 2426913079401922388,
2451699806267654588, 2506037868809622944, 2539177368085335242, 2585691867824138041, 2615916489656464006,
2629622650576085045, 2657709892891143999, 269267015073834956, 2978608649954009655, 2994749269564046800,
3002378570488766614, 3121394566209939036, 3141180417712080576, 3154166401188000671, 3295643071378191280,
3367253256599281972, 3373165743103532333, 3410319489097671220, 341173714064808701, 35319237546365581,
358168701228763255, 3695798865479120909, 3784007090181772945, 3859215587032554422, 388414320421460408,
3958884439975359895, 4116791292047135794, 4157923384507172767, 4177479403175856644, 4192843956567907187,
4237835213950914050, 4283455088657318777, 4437755738583404917, 4464954521637384793, 4479350931740982844,
448168435071015674, 4483893029099418629, 4495279466361191724, 4626894508809678069, 4913067912592229158,
5055224510936602124, 5146899889661568824, 5333227006937586923, 5344713819757688419, 5354664536118782383,
5407856766112305920, 5524147618224097934, 5530799914723849578, 5594217902784391863, 5740424123774088150,
5773813700926694122, 5797583289244413653, 596003237801815323, 6051342740892152639, 6053457919557951562,
6063733117952809262, 6188792405330710479, 6215325692940778409, 6255014464234787541, 6307265543356112469,
6331411861541807584, 6428394292228979654, 6440759262443849168, 6519598944774579329, 6559591672744555181,
6637797466881599715, 6662404025200321225, 6668777379548711499, 6828109123001551836, 6861072227483193626,
6876892389046410599, 6982428538201939129, 7004630263067699041, 703529510861716420, 7035478043169974728,
713068522342900631, 7143695289303246630, 7149047549904215441, 7302396691607427712, 7315761441890909840,
7410347447605570455, 754235745869667971, 7544573809574793144, 7586097788690287546, 7597038064500920869,
7614889355576709597, 7624148922826428440, 7627724472890172915, 7896536432019982357, 8070449930340402428,
8100741102346464407, 8206476157175887120, 8277001814877694995, 8323125797400536081, 8331290716255307824,
8338939509697372178, 834182466138639995, 8388253326997854443, 8414277759665359090, 8670923111306643765,
8769994749009616915, 8770706585661243637, 8812518846508548955, 8847966629758817746, 8906298870427014138,
8987438727642595319, 9050092026849126476, 933979495781858352, 962613924858074837, 985754492835172659]
for /10.0.1.32"}
{"time":"2016-02-05T11:54:32.138Z","level":"INFO","msg":"Deleting any stored hints for /10.0.1.32"}
{"time":"2016-02-05T11:54:32.161Z","level":"INFO","msg":"Handshaking version with /10.0.1.32"}
{noformat}

> Decommissioned nodes can remain in gossip
> -----------------------------------------
>
>                 Key: CASSANDRA-10371
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10371
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Distributed Metadata
>            Reporter: Brandon Williams
>            Assignee: Joel Knighton
>            Priority: Minor
>
> This may apply to other dead states as well.  Dead states should be expired after 3 days.
 In the case of decom we attach a timestamp to let the other nodes know when it should be
expired.  It has been observed that sometimes a subset of nodes in the cluster never expire
the state, and through heap analysis of these nodes it is revealed that the epstate.isAlive
check returns true when it should return false, which would allow the state to be evicted.
 This may have been affected by CASSANDRA-8336.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message