Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9E1521737C for ; Tue, 1 Sep 2015 14:23:17 +0000 (UTC) Received: (qmail 70715 invoked by uid 500); 1 Sep 2015 14:23:12 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 70614 invoked by uid 500); 1 Sep 2015 14:23:11 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 70561 invoked by uid 99); 1 Sep 2015 14:23:11 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2015 14:23:11 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 6DAD8C0C9B for ; Tue, 1 Sep 2015 14:23:11 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.898 X-Spam-Level: **** X-Spam-Status: No, score=4.898 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, KAM_BADIPHTTP=2, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id GozAsxBBWA9b for ; Tue, 1 Sep 2015 14:23:08 +0000 (UTC) Received: from mail-ob0-f176.google.com (mail-ob0-f176.google.com [209.85.214.176]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 8B1E024A72 for ; Tue, 1 Sep 2015 14:23:08 +0000 (UTC) Received: by obqa2 with SMTP id a2so268749obq.3 for ; Tue, 01 Sep 2015 07:23:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=yu2t1i8HsoiRyZxz3yzmB7VXZrijLYaITpAXppwr/VY=; b=02hSyrcPGcqJf92lyh1v2W0YAmDf1KWasM0NO2MRmoRAKvwL7MN1HmQzPAfn4IOKDY aAxaEqzm3ExQpun9PqfxGz6dYcOH3O0u8mRNwxe/Tfka288GAM+RdO8k8XGh5ps52+S3 4iorbuWlqVd9yLxtunT1pLCFqiVFlDQPZJSQUuPWn2jKTggRmKIeqd21DVKI0jZLfxX+ 4j3hHkuk2AgdVh3/vXTbQTfBS7gtIFLxa9t0+AOUjiJSYhf4Ij8Y7wDGoTRAw7LYiSgh Ps2+aT3Hfqc/wkCOVykihggUdBmKXucxW08Wo7zckWg8SclGThm4MpJVrNLEJ62dgtrN panw== X-Received: by 10.60.35.98 with SMTP id g2mr17176614oej.6.1441117387952; Tue, 01 Sep 2015 07:23:07 -0700 (PDT) MIME-Version: 1.0 Received: by 10.182.45.233 with HTTP; Tue, 1 Sep 2015 07:22:48 -0700 (PDT) In-Reply-To: References: From: =?UTF-8?B?5pu55b+X5a+M?= Date: Tue, 1 Sep 2015 22:22:48 +0800 Message-ID: Subject: Re: abnormal log after remove a node To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=089e01182c10fd180d051eb04913 --089e01182c10fd180d051eb04913 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Just restart all of the c* node -------------------------------------- Ranger Tsao 2015-08-25 18:17 GMT+08:00 Alain RODRIGUEZ : > Hi, I am facing the same issue on 2.0.16. > > Did you solve this ? How ? > > I plan to try a rolling restart and see if gossip state recover from this= . > > C*heers, > > Alain > > 2015-06-19 11:40 GMT+02:00 =E6=9B=B9=E5=BF=97=E5=AF=8C : > >> I have a C* 2.1.5 with 24 nodes.A few days ago ,I have remove a node fro= m >> this cluster using nodetool decommission. >> >> But tody I find some log like this: >> >> INFO [GossipStage:1] 2015-06-19 17:38:05,616 Gossiper.java:968 - >> InetAddress /172.19.105.41 is now DOWN >> INFO [GossipStage:1] 2015-06-19 17:38:05,617 StorageService.java:1885 - >> Removing tokens [-1014432261309809702, -1055322450438958612, >> -1120728727235087395, -1191392141261832305, -1203676771883970142, >> -1215563040745505837, -1215648909329054362, -1269531760567530381, >> -1278047879489577908, -1313427877031136549, -1342822572958042617, >> -1350792764922315814, -1383390744017639599, -139000372807970456, >> -140827955201469664, -1631551789771606023, -1633789813430312609, >> -1795528665156349205, -1836619444785023397, -1879127294549041822, >> -1962337787208890426, -2022309807234530256, -2033402140526360327, >> -2089413865145942100, -210961549458416802, -2148530352195763113, >> -2184481573787758786, -2222610790268720205, -2340762266634834427, >> -2513416003567685694, -2520971378752190013, -2596695976621541808, >> -2620636796023437199, -2640378596436678113, -2679143017361311011, >> -2721176590519112233, -2749213392354746126, -279267896827516626, >> -2872377759991294853, -2904711688111888325, -290489381926812623, >> -3000574339499272616, -301428600802598523, -3019280155316984595, >> -3024451041907074275, -3056898917375012425, -3161300347260716852, >> -3166392383659271772, -3327634380871627036, -3530685865340274372, >> -3563112657791369745, -366930313427781469, -3729582520450700795, >> -3901838244986519991, -4065326606010524312, -4174346928341550117, >> -4184239233207315432, -4204369933734181327, -4206479093137814808, >> -421410317165821100, -4311166118017934135, -4407123461118340117, >> -4466364858622123151, -4466939645485100087, -448955147512581975, >> -4587780638857304626, -4649897584350376674, -4674234125365755024 >> , -4833801201210885896, -4857586579802212277, -4868896650650107463, >> -4980063310159547694, -4983471821416248610, -4992846054037653676, >> -5026994389965137674, -514302500353679181 >> 0, -5198414516309928594, -5245363745777287346, -5346838390293957674, >> -5374413419545696184, -5427881744040857637, -5453876964430787287, >> -5491923669475601173, -55219734138599212 >> 6, -5523011502670737422, -5537121117160410549, -5557015938925208697, >> -5572489682738121748, -5745899409803353484, -5771239101488682535, >> -5893479791287484099, -59766730414807540 >> 44, -6014643892406938367, -6086002438656595783, -6129360679394503700, >> -6224240257573911174, -6290393495130499466, -6378712056928268929, >> -6430306056990093461, -6800188263839065 >> 013, -6912720411187525051, -7160327814305587432, -7175004328733776324, >> -7272070430660252577, -7307945744786025148, -742448651973108101, >> -7539255117639002578, -7657460716997978 >> 94, -7846698077070579798, -7870621904906244395, -7900841391761900719, >> -7918145426423910061, -7936795453892692473, -8070255024778921411, >> -8086888710627677669, -8124855925323654 >> 631, -8175270408138820500, -8271197636596881168, -8336685710406477123, >> -8466220397076441627, -8534337908154758270, -8550484400487603561, >> -862246738021989870, -8727219287242892 >> 185, -8895705475282612927, -8921801772904834063, -9057266752652143883, >> -9059183540698454288, -9067986437682229598, -9148183367896132028, >> -962208188860606543, 10859447725819218 >> 30, 1189775396643491793, 1253728955879686947, 1389982523380382228, >> 1429632314664544045, 143610053770130548, 150118120072602242, >> 1575692041584712198, 1624575905722628764, 17894 >> 76212785155173, 1995296121962835019, 2041217364870030239, >> 2120277336231792146, 2124445736743406711, 2154979704292433983, >> 2340726755918680765, 23481654796845972, 23620268084352 >> 24407, 2366144489007464626, 2381492708106933027, 2398868971489617398, >> 2427315953339163528, 2433999003913998534, 2633074510238705620, >> 266659839023809792, 2677817641360639089, 2 >> 719725410894526151, 2751925111749406683, 2815703589803785617, >> 3041515796379693113, 3044903149214270978, 3094954503756703989, >> 3243933267690865263, 3246086646486800371, 33270068 >> 97333869434, 3393657685587750192, 3395065499228709345, >> 3426126123948029459, 3500469615600510698, 3644011364716880512, >> 3693249207133187620, 3776164494954636918, 385555780676797 >> 8035, 3872151295451662867, 3937077827707223414, 4041082935346014761, >> 4060208918173638435, 4086747843759164940, 4165638694482690057, >> 4203996339238989224, 4220155275330961826, 4 >> 366784953339236686, 4390116924352514616, 4391225331964772681, >> 4392419346255765958, 4448400054980766409, 4463335839328115373, >> 4547306976104362915, 4588174843388248100, 48438580 >> 67983993745, 4912719175808770608, 499628843707992459, >> 5004392861473086088, 5021047773702107258, 510226752691159107, >> 5109551630357971118, 5157669927051121583, 51627694176199618 >> 24, 5238710860488961530, 5245958115092331518, 5302459768185143407, >> 5373077323749320571, 5445982956737768774, 5526076427753104565, >> 5531878975169972758, 5590672474842108747, 561 >> 8238086143944892, 5645763748154253201, 5648082473497629258, >> 5799608283794045232, 5968931466409317704, 6080339666926312644, >> 6222992739052178144, 6329332485451402638, 6372655016 >> 269952381, 6450034006536146156, 6526155371478962031, 6548600217631455803= , >> 6591556769219166621, 6658093966781219462, 6673583970197595202, >> 6721108087363914888, 67250010230161815 >> 66, 6755422648396704654, 6771784849502850301, 6894457688282530581, >> 6935906357620278103, 7290035612348735845, 7391177400231418753, >> 7392159536582094629, 7416025635188268574, 777 >> 2839976879697009, 7779944872726958967, 7813009726137267594, >> 7828693936020907346, 783798051009400395, 794321710728449974, >> 7969419520390622697, 8032162688041296427, 813984063616 >> 8254157, 8246865194299129430, 8318906537193659373, 842843196067279262, >> 8540875087395336317, 8666350659048928104, 8697352861510764965, >> 8801847806932416388, 8828239026404690350, >> 8874223097578484531, 8880048920270623806, 8882247680076975559, >> 89230101686533845, 8977307252904558369, 9037550178585102557, >> 9199020538928645866] for /172.19.105.41 >> >> >> ant this log info a few minutes one time >> >> >> -------------------------------------- >> Ranger Tsao >> > > --089e01182c10fd180d051eb04913 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Just restart all of the c* node

--------------------------------------
=
Ranger Tsao=

2015-08-25 18:17 GMT+08:00 Alain RODRIGUEZ <= span dir=3D"ltr"><arodrime@gmail.com>:
Hi, I am facing the same issue on 2.0.16.

D= id you solve this ? How ?

I plan to try a rolling = restart and see if gossip state recover from this.

=
C*heers,

Alain

2015-06-19 11:40 GMT+02:00 =E6=9B=B9=E5=BF=97=E5=AF=8C <cao.zhifu@gm= ail.com>:
=
I have a C* 2.1.5 with 24 nodes.A few days ago ,I have remove a node f= rom this cluster using nodetool=C2=A0decommission.<= /font>

But tody I find some log like this:

INFO =C2=A0[Go= ssipStage:1] 2015-06-19 17:38:05,616 Gossiper.java:968 - InetAddress /172.19.105.41 is now DOWN=
INFO =C2=A0[GossipStage:1] 2015-06-19 17:38:05,617 StorageServic= e.java:1885 - Removing tokens [-1014432261309809702, -1055322450438958612, = -1120728727235087395, -1191392141261832305, -1203676771883970142, -12155630= 40745505837, -1215648909329054362, -1269531760567530381, -12780478794895779= 08, -1313427877031136549, -1342822572958042617, -1350792764922315814, -1383= 390744017639599, -139000372807970456, -140827955201469664, -163155178977160= 6023, -1633789813430312609, -1795528665156349205, -1836619444785023397, -18= 79127294549041822, -1962337787208890426, -2022309807234530256, -20334021405= 26360327, -2089413865145942100, -210961549458416802, -2148530352195763113, = -2184481573787758786, -2222610790268720205, -2340762266634834427, -25134160= 03567685694, -2520971378752190013, -2596695976621541808, -26206367960234371= 99, -2640378596436678113, -2679143017361311011, -2721176590519112233, -2749= 213392354746126, -279267896827516626, -2872377759991294853, -29047116881118= 88325, -290489381926812623, -3000574339499272616, -301428600802598523, -301= 9280155316984595, -3024451041907074275, -3056898917375012425, -316130034726= 0716852, -3166392383659271772, -3327634380871627036, -3530685865340274372, = -3563112657791369745, -366930313427781469, -3729582520450700795, -390183824= 4986519991, -4065326606010524312, -4174346928341550117, -418423923320731543= 2, -4204369933734181327, -4206479093137814808, -421410317165821100, -431116= 6118017934135, -4407123461118340117, -4466364858622123151, -446693964548510= 0087, -448955147512581975, -4587780638857304626, -4649897584350376674, -467= 4234125365755024
, -4833801201210885896, -4857586579802212277, -4= 868896650650107463, -4980063310159547694, -4983471821416248610, -4992846054= 037653676, -5026994389965137674, -514302500353679181
0, -51984145= 16309928594, -5245363745777287346, -5346838390293957674, -53744134195456961= 84, -5427881744040857637, -5453876964430787287, -5491923669475601173, -5521= 9734138599212
6, -5523011502670737422, -5537121117160410549, -555= 7015938925208697, -5572489682738121748, -5745899409803353484, -577123910148= 8682535, -5893479791287484099, -59766730414807540
44, -6014643892= 406938367, -6086002438656595783, -6129360679394503700, -6224240257573911174= , -6290393495130499466, -6378712056928268929, -6430306056990093461, -680018= 8263839065
013, -6912720411187525051, -7160327814305587432, -7175= 004328733776324, -7272070430660252577, -7307945744786025148, -7424486519731= 08101, -7539255117639002578, -7657460716997978
94, -7846698077070= 579798, -7870621904906244395, -7900841391761900719, -7918145426423910061, -= 7936795453892692473, -8070255024778921411, -8086888710627677669, -812485592= 5323654
631, -8175270408138820500, -8271197636596881168, -8336685= 710406477123, -8466220397076441627, -8534337908154758270, -8550484400487603= 561, -862246738021989870, -8727219287242892
185, -889570547528261= 2927, -8921801772904834063, -9057266752652143883, -9059183540698454288, -90= 67986437682229598, -9148183367896132028, -962208188860606543, 1085944772581= 9218
30, 1189775396643491793, 1253728955879686947, 13899825233803= 82228, 1429632314664544045, 143610053770130548, 150118120072602242, 1575692= 041584712198, 1624575905722628764, 17894
76212785155173, 19952961= 21962835019, 2041217364870030239, 2120277336231792146, 2124445736743406711,= 2154979704292433983, 2340726755918680765, 23481654796845972, 2362026808435= 2
24407, 2366144489007464626, 2381492708106933027, 23988689714896= 17398, 2427315953339163528, 2433999003913998534, 2633074510238705620, 26665= 9839023809792, 2677817641360639089, 2
719725410894526151, 2751925= 111749406683, 2815703589803785617, 3041515796379693113, 3044903149214270978= , 3094954503756703989, 3243933267690865263, 3246086646486800371, 33270068
97333869434, 3393657685587750192, 3395065499228709345, 34261261239= 48029459, 3500469615600510698, 3644011364716880512, 3693249207133187620, 37= 76164494954636918, 385555780676797
8035, 3872151295451662867, 393= 7077827707223414, 4041082935346014761, 4060208918173638435, 408674784375916= 4940, 4165638694482690057, 4203996339238989224, 4220155275330961826, 4
366784953339236686, 4390116924352514616, 4391225331964772681, 4392419= 346255765958, 4448400054980766409, 4463335839328115373, 4547306976104362915= , 4588174843388248100, 48438580
67983993745, 4912719175808770608,= 499628843707992459, 5004392861473086088, 5021047773702107258, 510226752691= 159107, 5109551630357971118, 5157669927051121583, 51627694176199618
24, 5238710860488961530, 5245958115092331518, 5302459768185143407, 53730= 77323749320571, 5445982956737768774, 5526076427753104565, 55318789751699727= 58, 5590672474842108747, 561
8238086143944892, 564576374815425320= 1, 5648082473497629258, 5799608283794045232, 5968931466409317704, 608033966= 6926312644, 6222992739052178144, 6329332485451402638, 6372655016
= 269952381, 6450034006536146156, 6526155371478962031, 6548600217631455803, 6= 591556769219166621, 6658093966781219462, 6673583970197595202, 6721108087363= 914888, 67250010230161815
66, 6755422648396704654, 67717848495028= 50301, 6894457688282530581, 6935906357620278103, 7290035612348735845, 73911= 77400231418753, 7392159536582094629, 7416025635188268574, 777
283= 9976879697009, 7779944872726958967, 7813009726137267594, 782869393602090734= 6, 783798051009400395, 794321710728449974, 7969419520390622697, 80321626880= 41296427, 813984063616
8254157, 8246865194299129430, 831890653719= 3659373, 842843196067279262, 8540875087395336317, 8666350659048928104, 8697= 352861510764965, 8801847806932416388, 8828239026404690350,
=C2=A0= 8874223097578484531, 8880048920270623806, 8882247680076975559, 892301016865= 33845, 8977307252904558369, 9037550178585102557, 9199020538928645866] for /= 172.19.105.41
<= div>

ant this log info a few minutes one time=
=C2=A0 =C2=A0

--------------------------------------
Ranger T= sao


--089e01182c10fd180d051eb04913--