Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAEDUwd2HD+AOoK=VgKF8Y7Zkj57c1LhER5Ht0Sk_G2RFupwqVQ@mail.gmail.com>
References: 
 <CA+OtsXtb5wPtp4GJjhs7GNbR6wt1bC5SSKBUFPnrRnTw3S1X_w@mail.gmail.com>
 <CACjDk5=VcVNDnTW41AhQVj1QkwS90D6Veo+Q4FKTWgjaWDY7sQ@mail.gmail.com>
 <CAEDUwd2HD+AOoK=VgKF8Y7Zkj57c1LhER5Ht0Sk_G2RFupwqVQ@mail.gmail.com>
From: Analia Lorenzatto <analialorenzatto@gmail.com>
Date: Tue, 12 May 2015 16:10:32 -0300
Message-ID: 
 <CA+OtsXv9ewS6LJ-p_KpWg5BXC28W8PPfEwBWGqZiw3riUs+=Jg@mail.gmail.com>
Subject: Re: Unexpected behavior after adding successffully a new node
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=047d7b5d2d32daa2900515e74089

--047d7b5d2d32daa2900515e74089
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Just in case I want to clarify that after bootstrapping the third node, it
got data and seemed to be working fine.  But it was the last night when
 the cluster started behaving in a weird way.  The last node (successfully
added last week) were being reported up and down all the time. After
restarting cassandra service, the node got empty but reported as Up and
Normal.

Also, the amount of data reported by nodetool on the other nodes is much
more than the amount already used in the device, is that normal?

root@10.0.0.a:~# df -h /mnt
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdb1      745G  163G  583G  22% /mnt/cassandra

root@:10.0.0.b~$ df -h /mnt
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdb1      745G  196G  550G  27% /mnt/cassandra


# nodetool status
Datacenter: us-east
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Status=3DUp/Down
|/ State=3DNormal/Leaving/Joining/Moving
--  Address     Load       Tokens  Owns (effective)  Host ID
                Rack
UN  10.0.0.a  391.34 GB  256     66.7%
eed9e9f5-f279-4b2f-b521-c056cbf65b52  1c
UN  10.0.0.b  383.38 GB  256     68.3%
19492c26-4458-4a0b-af04-72e0aab6598e  1c

Thanks in advance!


On Tue, May 12, 2015 at 2:35 PM, Robert Coli <rcoli@eventbrite.com> wrote:

> On Tue, May 12, 2015 at 9:59 AM, arun sirimalla <arunsirik@gmail.com>
> wrote:
>
>> Try running repair on node 3.
>>
>
> Mostly disagree. If a node is empty after a bootstrap, remove it and
> re-bootstrap it.
>
> =3DRob
>
>


--=20
Saludos / Regards.

Anal=C3=ADa Lorenzatto.

=E2=80=9CIt's possible to commit no errors and still lose. That is not weak=
ness.
That is life".  By Captain Jean-Luc Picard.

--047d7b5d2d32daa2900515e74089
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Just in case I want to clarify that after bootstrapping th=
e third node, it got data and seemed to be working fine.=C2=A0 But it was t=
he last night when =C2=A0the cluster started behaving in a weird way.=C2=A0=
 The last node (successfully added last week) were being reported up and do=
wn all the time. After restarting cassandra service, the node got empty but=
 reported as Up and Normal.<div><br></div><div>Also, the amount of data rep=
orted by nodetool on the other nodes is much more than the amount already u=
sed in the device, is that normal?</div><div><br></div><div><div>root@10.0.=
0.a:~# df -h /mnt</div><div>Filesystem =C2=A0 =C2=A0 =C2=A0Size =C2=A0Used =
Avail Use% Mounted on</div><div>/dev/xvdb1 =C2=A0 =C2=A0 =C2=A0745G =C2=A01=
63G =C2=A0583G =C2=A022% /mnt/cassandra</div></div><div><br></div><div><div=
>root@:10.0.0.b~$ df -h /mnt</div><div>Filesystem =C2=A0 =C2=A0 =C2=A0Size =
=C2=A0Used Avail Use% Mounted on</div><div>/dev/xvdb1 =C2=A0 =C2=A0 =C2=A07=
45G =C2=A0196G =C2=A0550G =C2=A027% /mnt/cassandra</div></div><div><br></di=
v><div><br></div><div><div># nodetool status</div><div>Datacenter: us-east<=
/div><div>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D</div><d=
iv>Status=3DUp/Down</div><div>|/ State=3DNormal/Leaving/Joining/Moving</div=
><div>-- =C2=A0Address =C2=A0 =C2=A0 Load =C2=A0 =C2=A0 =C2=A0 Tokens =C2=
=A0Owns (effective) =C2=A0Host ID =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Rack</div><=
div>UN =C2=A010.0.0.a =C2=A0391.34 GB =C2=A0256 =C2=A0 =C2=A0 66.7% =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 eed9e9f5-f279-4b2f-b521-c056cbf65b52 =C2=
=A01c</div><div>UN =C2=A010.0.0.b =C2=A0383.38 GB =C2=A0256 =C2=A0 =C2=A0 6=
8.3% =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 19492c26-4458-4a0b-af04-72e0=
aab6598e =C2=A01c</div></div><div><br></div><div>Thanks in advance!<br></di=
v><div><br></div><div><br></div><div><br></div></div><div class=3D"gmail_ex=
tra"><br><div class=3D"gmail_quote">On Tue, May 12, 2015 at 2:35 PM, Robert=
 Coli <span dir=3D"ltr">&lt;<a href=3D"mailto:rcoli@eventbrite.com" target=
=3D"_blank">rcoli@eventbrite.com</a>&gt;</span> wrote:<br><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex"><div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gma=
il_quote"><span class=3D"">On Tue, May 12, 2015 at 9:59 AM, arun sirimalla =
<span dir=3D"ltr">&lt;<a href=3D"mailto:arunsirik@gmail.com" target=3D"_bla=
nk">arunsirik@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail=
_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:=
1ex"><div dir=3D"ltr"><div>Try running repair on node 3.</div></div></block=
quote><div><br></div></span><div>Mostly disagree. If a node is empty after =
a bootstrap, remove it and re-bootstrap it.</div><div><br></div><div>=3DRob=
</div><div>=C2=A0</div></div></div></div>
</blockquote></div><br><br clear=3D"all"><div><br></div>-- <br><div class=
=3D"gmail_signature"><div dir=3D"ltr">Saludos / Regards.<div><br></div><div=
>Anal=C3=ADa Lorenzatto.<br><br>=E2=80=9CIt&#39;s possible to commit no err=
ors and still lose. That is not weakness.=C2=A0 That is life&quot;.=C2=A0 B=
y Captain Jean-Luc Picard.</div></div></div>
</div>

--047d7b5d2d32daa2900515e74089--