Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B6E0D1754F for ; Tue, 1 Sep 2015 14:52:42 +0000 (UTC) Received: (qmail 67286 invoked by uid 500); 1 Sep 2015 14:52:39 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 67264 invoked by uid 500); 1 Sep 2015 14:52:38 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 67246 invoked by uid 99); 1 Sep 2015 14:52:38 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2015 14:52:38 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 784441AB38E for ; Tue, 1 Sep 2015 14:52:38 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.879 X-Spam-Level: **** X-Spam-Status: No, score=4.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, KAM_BADIPHTTP=2, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id FwCj5gImhJRH for ; Tue, 1 Sep 2015 14:52:35 +0000 (UTC) Received: from mail-la0-f46.google.com (mail-la0-f46.google.com [209.85.215.46]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 3AF1F26457 for ; Tue, 1 Sep 2015 14:52:35 +0000 (UTC) Received: by lanb10 with SMTP id b10so1391444lan.3 for ; Tue, 01 Sep 2015 07:52:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=oTEnXfu04ooTaOVNnlLW/esl5CUF+oqGNDqkTpXu4bQ=; b=p5LcGmXgRVYMmXTZfc4sZcEAkJfdkeBq3uVxvYhEnpzPOp+lT3P1s8O27qF627PaOH m573AxmmznzaSiSJE5dC8qL7jhW6CRAPhSpwjzZkysoD0uI+J4+9h7K6lysccRAnX4gZ 1GdPpHFYJVZbMN6GGtVs/fHmMRLiW44WY7XmWLyPbK7F03tQMR8WieRcLTdDWyDkMuwc NaXoWRGeEMfxUJvkkeaP07p3/lL16NqmctiwP8kGUWzqmIQFYIjuLHJL8q/yNXsjqMRV qrFNQZGrziwMaxwfqWfVNk0w2DF/bt5OlBPe9mSPEeRjhhAwcx6EXwdRxx+/BL/YL3Iq cO9Q== X-Received: by 10.152.28.229 with SMTP id e5mr13339236lah.110.1441119153568; Tue, 01 Sep 2015 07:52:33 -0700 (PDT) MIME-Version: 1.0 Received: by 10.112.208.104 with HTTP; Tue, 1 Sep 2015 07:52:13 -0700 (PDT) In-Reply-To: References: From: Alain RODRIGUEZ Date: Tue, 1 Sep 2015 16:52:13 +0200 Message-ID: Subject: Re: abnormal log after remove a node To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=089e0160b4523a452d051eb0b320 --089e0160b4523a452d051eb0b320 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi, I finally did the exact same thing before receiving the answer. I guess this will remain FTR :). Thanks though ! Alain 2015-09-01 16:22 GMT+02:00 =E6=9B=B9=E5=BF=97=E5=AF=8C : > Just restart all of the c* node > > -------------------------------------- > Ranger Tsao > > 2015-08-25 18:17 GMT+08:00 Alain RODRIGUEZ : > >> Hi, I am facing the same issue on 2.0.16. >> >> Did you solve this ? How ? >> >> I plan to try a rolling restart and see if gossip state recover from thi= s. >> >> C*heers, >> >> Alain >> >> 2015-06-19 11:40 GMT+02:00 =E6=9B=B9=E5=BF=97=E5=AF=8C : >> >>> I have a C* 2.1.5 with 24 nodes.A few days ago ,I have remove a node >>> from this cluster using nodetool decommission. >>> >>> But tody I find some log like this: >>> >>> INFO [GossipStage:1] 2015-06-19 17:38:05,616 Gossiper.java:968 - >>> InetAddress /172.19.105.41 is now DOWN >>> INFO [GossipStage:1] 2015-06-19 17:38:05,617 StorageService.java:1885 = - >>> Removing tokens [-1014432261309809702, -1055322450438958612, >>> -1120728727235087395, -1191392141261832305, -1203676771883970142, >>> -1215563040745505837, -1215648909329054362, -1269531760567530381, >>> -1278047879489577908, -1313427877031136549, -1342822572958042617, >>> -1350792764922315814, -1383390744017639599, -139000372807970456, >>> -140827955201469664, -1631551789771606023, -1633789813430312609, >>> -1795528665156349205, -1836619444785023397, -1879127294549041822, >>> -1962337787208890426, -2022309807234530256, -2033402140526360327, >>> -2089413865145942100, -210961549458416802, -2148530352195763113, >>> -2184481573787758786, -2222610790268720205, -2340762266634834427, >>> -2513416003567685694, -2520971378752190013, -2596695976621541808, >>> -2620636796023437199, -2640378596436678113, -2679143017361311011, >>> -2721176590519112233, -2749213392354746126, -279267896827516626, >>> -2872377759991294853, -2904711688111888325, -290489381926812623, >>> -3000574339499272616, -301428600802598523, -3019280155316984595, >>> -3024451041907074275, -3056898917375012425, -3161300347260716852, >>> -3166392383659271772, -3327634380871627036, -3530685865340274372, >>> -3563112657791369745, -366930313427781469, -3729582520450700795, >>> -3901838244986519991, -4065326606010524312, -4174346928341550117, >>> -4184239233207315432, -4204369933734181327, -4206479093137814808, >>> -421410317165821100, -4311166118017934135, -4407123461118340117, >>> -4466364858622123151, -4466939645485100087, -448955147512581975, >>> -4587780638857304626, -4649897584350376674, -4674234125365755024 >>> , -4833801201210885896, -4857586579802212277, -4868896650650107463, >>> -4980063310159547694, -4983471821416248610, -4992846054037653676, >>> -5026994389965137674, -514302500353679181 >>> 0, -5198414516309928594, -5245363745777287346, -5346838390293957674, >>> -5374413419545696184, -5427881744040857637, -5453876964430787287, >>> -5491923669475601173, -55219734138599212 >>> 6, -5523011502670737422, -5537121117160410549, -5557015938925208697, >>> -5572489682738121748, -5745899409803353484, -5771239101488682535, >>> -5893479791287484099, -59766730414807540 >>> 44, -6014643892406938367, -6086002438656595783, -6129360679394503700, >>> -6224240257573911174, -6290393495130499466, -6378712056928268929, >>> -6430306056990093461, -6800188263839065 >>> 013, -6912720411187525051, -7160327814305587432, -7175004328733776324, >>> -7272070430660252577, -7307945744786025148, -742448651973108101, >>> -7539255117639002578, -7657460716997978 >>> 94, -7846698077070579798, -7870621904906244395, -7900841391761900719, >>> -7918145426423910061, -7936795453892692473, -8070255024778921411, >>> -8086888710627677669, -8124855925323654 >>> 631, -8175270408138820500, -8271197636596881168, -8336685710406477123, >>> -8466220397076441627, -8534337908154758270, -8550484400487603561, >>> -862246738021989870, -8727219287242892 >>> 185, -8895705475282612927, -8921801772904834063, -9057266752652143883, >>> -9059183540698454288, -9067986437682229598, -9148183367896132028, >>> -962208188860606543, 10859447725819218 >>> 30, 1189775396643491793, 1253728955879686947, 1389982523380382228, >>> 1429632314664544045, 143610053770130548, 150118120072602242, >>> 1575692041584712198, 1624575905722628764, 17894 >>> 76212785155173, 1995296121962835019, 2041217364870030239, >>> 2120277336231792146, 2124445736743406711, 2154979704292433983, >>> 2340726755918680765, 23481654796845972, 23620268084352 >>> 24407, 2366144489007464626, 2381492708106933027, 2398868971489617398, >>> 2427315953339163528, 2433999003913998534, 2633074510238705620, >>> 266659839023809792, 2677817641360639089, 2 >>> 719725410894526151, 2751925111749406683, 2815703589803785617, >>> 3041515796379693113, 3044903149214270978, 3094954503756703989, >>> 3243933267690865263, 3246086646486800371, 33270068 >>> 97333869434, 3393657685587750192, 3395065499228709345, >>> 3426126123948029459, 3500469615600510698, 3644011364716880512, >>> 3693249207133187620, 3776164494954636918, 385555780676797 >>> 8035, 3872151295451662867, 3937077827707223414, 4041082935346014761, >>> 4060208918173638435, 4086747843759164940, 4165638694482690057, >>> 4203996339238989224, 4220155275330961826, 4 >>> 366784953339236686, 4390116924352514616, 4391225331964772681, >>> 4392419346255765958, 4448400054980766409, 4463335839328115373, >>> 4547306976104362915, 4588174843388248100, 48438580 >>> 67983993745, 4912719175808770608, 499628843707992459, >>> 5004392861473086088, 5021047773702107258, 510226752691159107, >>> 5109551630357971118, 5157669927051121583, 51627694176199618 >>> 24, 5238710860488961530, 5245958115092331518, 5302459768185143407, >>> 5373077323749320571, 5445982956737768774, 5526076427753104565, >>> 5531878975169972758, 5590672474842108747, 561 >>> 8238086143944892, 5645763748154253201, 5648082473497629258, >>> 5799608283794045232, 5968931466409317704, 6080339666926312644, >>> 6222992739052178144, 6329332485451402638, 6372655016 >>> 269952381, 6450034006536146156, 6526155371478962031, >>> 6548600217631455803, 6591556769219166621, 6658093966781219462, >>> 6673583970197595202, 6721108087363914888, 67250010230161815 >>> 66, 6755422648396704654, 6771784849502850301, 6894457688282530581, >>> 6935906357620278103, 7290035612348735845, 7391177400231418753, >>> 7392159536582094629, 7416025635188268574, 777 >>> 2839976879697009, 7779944872726958967, 7813009726137267594, >>> 7828693936020907346, 783798051009400395, 794321710728449974, >>> 7969419520390622697, 8032162688041296427, 813984063616 >>> 8254157, 8246865194299129430, 8318906537193659373, 842843196067279262, >>> 8540875087395336317, 8666350659048928104, 8697352861510764965, >>> 8801847806932416388, 8828239026404690350, >>> 8874223097578484531, 8880048920270623806, 8882247680076975559, >>> 89230101686533845, 8977307252904558369, 9037550178585102557, >>> 9199020538928645866] for /172.19.105.41 >>> >>> >>> ant this log info a few minutes one time >>> >>> >>> -------------------------------------- >>> Ranger Tsao >>> >> >> > --089e0160b4523a452d051eb0b320 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi,

I finally did the exact same thing = before receiving the answer.

I guess this will rem= ain FTR :).

Thanks though !

Alain


2015-09-01 16:22 GMT+02:00 =E6=9B=B9=E5=BF=97=E5=AF=8C <cao.zhifu@gmail.com>:
Just restart all of the c* node

------------= --------------------------
Ranger Tsao

2015-08-25 18:17 GMT+08:00 Alain RODRIGUEZ <= span dir=3D"ltr"><arodrime@gmail.com>:
Hi, I am facing the same issue on 2.0.16.

D= id you solve this ? How ?

I plan to try a rolling = restart and see if gossip state recover from this.

=
C*heers,

Alain

2015-06-19 11:40 GMT+02:00= =E6=9B=B9=E5=BF=97=E5=AF=8C <cao.zhifu@gmail.com>:
I have a C* 2.1.5 with 24= nodes.A few days ago ,I have remove a node from this cluster using nodetool=C2=A0decommission.

But tody I find = some log like this:

INFO =C2=A0[GossipStage:1] 2015-06-19 17:38= :05,616 Gossiper.java:968 - InetAddress /172.19.105.41 is now DOWN
INFO =C2=A0[Gossip= Stage:1] 2015-06-19 17:38:05,617 StorageService.java:1885 - Removing tokens= [-1014432261309809702, -1055322450438958612, -1120728727235087395, -119139= 2141261832305, -1203676771883970142, -1215563040745505837, -121564890932905= 4362, -1269531760567530381, -1278047879489577908, -1313427877031136549, -13= 42822572958042617, -1350792764922315814, -1383390744017639599, -13900037280= 7970456, -140827955201469664, -1631551789771606023, -1633789813430312609, -= 1795528665156349205, -1836619444785023397, -1879127294549041822, -196233778= 7208890426, -2022309807234530256, -2033402140526360327, -208941386514594210= 0, -210961549458416802, -2148530352195763113, -2184481573787758786, -222261= 0790268720205, -2340762266634834427, -2513416003567685694, -252097137875219= 0013, -2596695976621541808, -2620636796023437199, -2640378596436678113, -26= 79143017361311011, -2721176590519112233, -2749213392354746126, -27926789682= 7516626, -2872377759991294853, -2904711688111888325, -290489381926812623, -= 3000574339499272616, -301428600802598523, -3019280155316984595, -3024451041= 907074275, -3056898917375012425, -3161300347260716852, -3166392383659271772= , -3327634380871627036, -3530685865340274372, -3563112657791369745, -366930= 313427781469, -3729582520450700795, -3901838244986519991, -4065326606010524= 312, -4174346928341550117, -4184239233207315432, -4204369933734181327, -420= 6479093137814808, -421410317165821100, -4311166118017934135, -4407123461118= 340117, -4466364858622123151, -4466939645485100087, -448955147512581975, -4= 587780638857304626, -4649897584350376674, -4674234125365755024
, = -4833801201210885896, -4857586579802212277, -4868896650650107463, -49800633= 10159547694, -4983471821416248610, -4992846054037653676, -50269943899651376= 74, -514302500353679181
0, -5198414516309928594, -524536374577728= 7346, -5346838390293957674, -5374413419545696184, -5427881744040857637, -54= 53876964430787287, -5491923669475601173, -55219734138599212
6, -5= 523011502670737422, -5537121117160410549, -5557015938925208697, -5572489682= 738121748, -5745899409803353484, -5771239101488682535, -5893479791287484099= , -59766730414807540
44, -6014643892406938367, -60860024386565957= 83, -6129360679394503700, -6224240257573911174, -6290393495130499466, -6378= 712056928268929, -6430306056990093461, -6800188263839065
013, -69= 12720411187525051, -7160327814305587432, -7175004328733776324, -72720704306= 60252577, -7307945744786025148, -742448651973108101, -7539255117639002578, = -7657460716997978
94, -7846698077070579798, -7870621904906244395,= -7900841391761900719, -7918145426423910061, -7936795453892692473, -8070255= 024778921411, -8086888710627677669, -8124855925323654
631, -81752= 70408138820500, -8271197636596881168, -8336685710406477123, -84662203970764= 41627, -8534337908154758270, -8550484400487603561, -862246738021989870, -87= 27219287242892
185, -8895705475282612927, -8921801772904834063, -= 9057266752652143883, -9059183540698454288, -9067986437682229598, -914818336= 7896132028, -962208188860606543, 10859447725819218
30, 1189775396= 643491793, 1253728955879686947, 1389982523380382228, 1429632314664544045, 1= 43610053770130548, 150118120072602242, 1575692041584712198, 162457590572262= 8764, 17894
76212785155173, 1995296121962835019, 2041217364870030= 239, 2120277336231792146, 2124445736743406711, 2154979704292433983, 2340726= 755918680765, 23481654796845972, 23620268084352
24407, 2366144489= 007464626, 2381492708106933027, 2398868971489617398, 2427315953339163528, 2= 433999003913998534, 2633074510238705620, 266659839023809792, 26778176413606= 39089, 2
719725410894526151, 2751925111749406683, 281570358980378= 5617, 3041515796379693113, 3044903149214270978, 3094954503756703989, 324393= 3267690865263, 3246086646486800371, 33270068
97333869434, 3393657= 685587750192, 3395065499228709345, 3426126123948029459, 3500469615600510698= , 3644011364716880512, 3693249207133187620, 3776164494954636918, 3855557806= 76797
8035, 3872151295451662867, 3937077827707223414, 40410829353= 46014761, 4060208918173638435, 4086747843759164940, 4165638694482690057, 42= 03996339238989224, 4220155275330961826, 4
366784953339236686, 439= 0116924352514616, 4391225331964772681, 4392419346255765958, 444840005498076= 6409, 4463335839328115373, 4547306976104362915, 4588174843388248100, 484385= 80
67983993745, 4912719175808770608, 499628843707992459, 50043928= 61473086088, 5021047773702107258, 510226752691159107, 5109551630357971118, = 5157669927051121583, 51627694176199618
24, 5238710860488961530, 5= 245958115092331518, 5302459768185143407, 5373077323749320571, 5445982956737= 768774, 5526076427753104565, 5531878975169972758, 5590672474842108747, 561<= /div>
8238086143944892, 5645763748154253201, 5648082473497629258, 57996= 08283794045232, 5968931466409317704, 6080339666926312644, 62229927390521781= 44, 6329332485451402638, 6372655016
269952381, 645003400653614615= 6, 6526155371478962031, 6548600217631455803, 6591556769219166621, 665809396= 6781219462, 6673583970197595202, 6721108087363914888, 67250010230161815
66, 6755422648396704654, 6771784849502850301, 6894457688282530581, 6= 935906357620278103, 7290035612348735845, 7391177400231418753, 7392159536582= 094629, 7416025635188268574, 777
2839976879697009, 77799448727269= 58967, 7813009726137267594, 7828693936020907346, 783798051009400395, 794321= 710728449974, 7969419520390622697, 8032162688041296427, 813984063616
<= div>8254157, 8246865194299129430, 8318906537193659373, 842843196067279262, = 8540875087395336317, 8666350659048928104, 8697352861510764965, 880184780693= 2416388, 8828239026404690350,
=C2=A08874223097578484531, 88800489= 20270623806, 8882247680076975559, 89230101686533845, 8977307252904558369, 9= 037550178585102557, 9199020538928645866] for /172.19.105.41


=
ant this log info a few minutes one time
=C2=A0 =C2=A0=

---------= -----------------------------
Ranger Tsao
<= /div>



--089e0160b4523a452d051eb0b320--