Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 91E201814D for ; Tue, 15 Sep 2015 12:45:06 +0000 (UTC) Received: (qmail 13231 invoked by uid 500); 15 Sep 2015 12:44:48 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 13190 invoked by uid 500); 15 Sep 2015 12:44:48 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 13180 invoked by uid 99); 15 Sep 2015 12:44:48 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Sep 2015 12:44:48 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id A1A571A1A75 for ; Tue, 15 Sep 2015 12:44:47 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.9 X-Spam-Level: ** X-Spam-Status: No, score=2.9 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id CpOB6crH8kFk for ; Tue, 15 Sep 2015 12:44:42 +0000 (UTC) Received: from mail-ig0-f181.google.com (mail-ig0-f181.google.com [209.85.213.181]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id A5EF121270 for ; Tue, 15 Sep 2015 12:44:41 +0000 (UTC) Received: by igxx6 with SMTP id x6so12047060igx.1 for ; Tue, 15 Sep 2015 05:44:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=cEK9r0wcJistYgaQHZj7dGSYY8WG80gXBqus3gVvn4w=; b=ZHGtb7luPfQEA7DcggBzWij+D3yrqo6YbGHVCtFob+mvIPppEbKx5DV+3X98w3YTi5 kHFNnIzbuZw228XGuDVGMEXhIJQjmn0PS8E05IWo00LWzwPBeEZLwt6VvTQsGgWLPMAf lSy26663vayxWgkOTS8+Mxz9DXS/bktjGq+gL1KQsMzetTnlYKdot8hzmMBD1CsfuLnL IzJVKTQ8u3+XMWA0Cbtc+iNEys5xMDjt3mTbo1mBoR6VNk+FmCI7tBackJYM/oI/w7CL +4zwm1ZMnCiSDn6CDuR7WAmKQYWuQUUQBjszM3PonJ5KH8X4y0zG3iO5vVpc6n3nSr4W ZwHw== X-Received: by 10.50.111.79 with SMTP id ig15mr4597074igb.51.1442321080444; Tue, 15 Sep 2015 05:44:40 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.14.6 with HTTP; Tue, 15 Sep 2015 05:44:21 -0700 (PDT) In-Reply-To: References: From: Mark Greene Date: Tue, 15 Sep 2015 08:44:21 -0400 Message-ID: Subject: Re: Cassandra adding node issue (no UJ status) To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=089e01294c8ca6ee85051fc88b5f --089e01294c8ca6ee85051fc88b5f Content-Type: text/plain; charset=UTF-8 Hey Rock, I've seen this occur as well. I've come to learn that in some cases, like a network blip, the join can fail. There is usually something in the log to the effect of "Stream failed" When I encounter this issue, I make an attempt to bootstrap the new node again. If that doesn't help, I run a repair on the new node. On Tue, Sep 15, 2015 at 3:14 AM, Rock Zhang wrote: > Hi All, > > > Now I got a problem everything when add new node, the data is not balance > again, just add it as new node. > > > Originally I saw the UJ status, anybody experienced this kind of issue ? I > don't know what changed . > > > ubuntu@ip-172-31-15-242:/etc/cassandra$ nodetool status rawdata > > Datacenter: DC1 > > =============== > > Status=Up/Down > > |/ State=Normal/Leaving/Joining/Moving > > -- Address Load Tokens Owns (effective) Host ID > Rack > > UN 172.31.16.191 742.15 GB 256 21.1% > 08638815-b721-46c4-b77c-af08285226db RAC1 > > UN 172.31.30.145 774.42 GB 256 21.2% > bde31dd9-ff1d-4f2f-b28d-fe54d0531c51 RAC1 > > UN 172.31.6.79 592.9 GB 256 19.8% > 15795fca-5425-41cd-909c-c1756715442a RAC2 > > UN 172.31.27.186 674.42 GB 256 18.4% > 9685a476-1da7-4c6f-819e-dd4483e3345e RAC1 > > UN 172.31.7.31 642.47 GB 256 19.9% > f7c8c6fb-ab37-4124-ba1a-a9a1beaecc1b RAC1 > > *UN 172.31.15.242 37.4 MB 256 19.8% > c3eff010-9904-49a0-83cd-258dc5a98525 RAC1* > > UN 172.31.24.32 780.59 GB 256 20.1% > ffa58bd1-3188-440d-94c9-97166ee4b735 RAC1 > > *UN 172.31.3.40 80.75 GB 256 18.9% > 01ce3f96-ebc0-4128-9ec3-ddd1a9845d51 RAC1* > > UN 172.31.31.238 756.59 GB 256 19.9% > 82d34a3b-4f12-4874-816c-7d89a0535577 RAC1 > > UN 172.31.31.99 583.68 GB 256 20.8% > 2b10194f-23d2-4bdc-bcfa-7961a149cd11 RAC2 > > > Thanks > > Rock > --089e01294c8ca6ee85051fc88b5f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hey Rock,

I've seen this occur = as well. I've come to learn that in some cases, like a network blip, th= e join can fail. There is usually something in the log to the effect of &qu= ot;Stream failed"

When I encounter this issue, I make an = attempt to bootstrap the new node again. If that doesn't help, I run a = repair on the new node.

On Tue, Sep 15, 2015 at 3:14 AM, Rock Zhang <= ;rock@alohar.com&g= t; wrote:

Hi All,=C2=A0


Now I got a problem everything when = add new node, the data is not balance again, just add it as new node.=C2=A0=


Originally I saw the UJ = status, anybody experienced this kind of issue ? I don't know what chan= ged .


<= p style=3D"margin:0px;font-size:11px;font-family:Menlo">ubuntu@ip-172-31-15= -242:/etc/cassandra$ nodetool status rawdata

Datacenter: DC1

=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D

Status=3DUp/Down

|/ State=3DNormal/= Leaving/Joining/Moving

--=C2=A0 Address= =C2=A0 =C2=A0 =C2=A0 =C2=A0 Load =C2=A0 =C2=A0 =C2=A0 Tokens=C2=A0 Owns (ef= fective)=C2=A0 Host ID =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Rack

UN=C2=A0 172.31.16= .191=C2=A0 742.15 GB=C2=A0 256 =C2=A0 =C2=A0 21.1% =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 08638815-b721-46c4-b77c-af08285226db=C2=A0 RAC1

UN=C2=A0 172.31.30= .145=C2=A0 774.42 GB=C2=A0 256 =C2=A0 =C2=A0 21.2% =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 bde31dd9-ff1d-4f2f-b28d-fe54d0531c51=C2=A0 RAC1

UN=C2=A0 172.31.6.= 79=C2=A0 =C2=A0 592.9 GB =C2=A0 256 =C2=A0 =C2=A0 19.8% =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 15795fca-5425-41cd-909c-c1756715442a=C2=A0 RAC2

UN=C2=A0 172.31.27= .186=C2=A0 674.42 GB=C2=A0 256 =C2=A0 =C2=A0 18.4% =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 9685a476-1da7-4c6f-819e-dd4483e3345e=C2=A0 RAC1

UN=C2=A0 172.31.7.= 31=C2=A0 =C2=A0 642.47 GB=C2=A0 256 =C2=A0 =C2=A0 19.9% =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 f7c8c6fb-ab37-4124-ba1a-a9a1beaecc1b=C2=A0 RAC1

UN=C2=A0 172.31.15.242=C2=A0 37.4 MB=C2=A0 =C2=A0 256 =C2=A0 =C2= =A0 19.8% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 c3eff010-9904-49a0-83cd= -258dc5a98525=C2=A0 RAC1

UN=C2=A0 172.31.24= .32 =C2=A0 780.59 GB=C2=A0 256 =C2=A0 =C2=A0 20.1% =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 ffa58bd1-3188-440d-94c9-97166ee4b735=C2=A0 RAC1

UN=C2=A0 172.31= .3.40=C2=A0 =C2=A0 80.75 GB =C2=A0 256 =C2=A0 =C2=A0 18.9% =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 01ce3f96-ebc0-4128-9ec3-ddd1a9845d51=C2=A0 RAC1=

UN=C2=A0 172.31.31= .238=C2=A0 756.59 GB=C2=A0 256 =C2=A0 =C2=A0 19.9% =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 82d34a3b-4f12-4874-816c-7d89a0535577=C2=A0 RAC1

UN=C2=A0 172.31.31= .99 =C2=A0 583.68 GB=C2=A0 256 =C2=A0 =C2=A0 20.8% =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 2b10194f-23d2-4bdc-bcfa-7961a149cd11=C2=A0 RAC2


Thanks

<= font color=3D"#888888">

Rock=C2=A0


--089e01294c8ca6ee85051fc88b5f--