Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of cheng.ren@bloomreach.com
 designates 209.85.212.182 as permitted sender)
MIME-Version: 1.0
Date: Mon, 9 Feb 2015 18:40:35 -0800
Message-ID: 
 <CALRai9DmWe9tqZGGjWu5pRd+ufNdXCGfWd2CDJ5TVzqXzyyUVA@mail.gmail.com>
Subject: nodetool status shows large numbers of up nodes are down
From: Cheng Ren <cheng.ren@bloomreach.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=047d7bb0498ec035f3050eb2cf7b

--047d7bb0498ec035f3050eb2cf7b
Content-Type: text/plain; charset=UTF-8

Hi,
We have a two-dc cluster with 21 nodes and 27 nodes in each DC. Over the
past few months, we have seen nodetool status marks 4-8 nodes down while
they are actually functioning. Particularly today we noticed that running
nodetool status on some nodes shows higher number of nodes are down than
before while they are actually up and serving requests.
For example, on one node it shows 42 nodes are down.

phi_convict_threshold of all nodes are set as 12, and we are running
cassandra 2.0.4 on AWS EC2 machines.

Does anyone have recommendation on identifying the root cause of this? Will
this cause any consequences?

Thanks,
Cheng

--047d7bb0498ec035f3050eb2cf7b
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi,<div>We have a two-dc cluster with 21 nodes and 27 node=
s in each DC. Over the past few months, we have seen nodetool status marks =
4-8 nodes down while they are actually functioning. Particularly today we n=
oticed that running nodetool status on some nodes shows higher number of no=
des are down than before while they are actually up and serving requests.=
=C2=A0</div><div>For example, on one node it shows 42 nodes are down.</div>=
<div><br></div><div>phi_convict_threshold of all nodes are set as 12, and w=
e are running cassandra 2.0.4 on AWS EC2 machines.</div><div><br></div><div=
>Does anyone have recommendation on identifying the root cause of this? Wil=
l this cause any consequences?</div><div><br></div><div>Thanks,</div><div>C=
heng=C2=A0</div></div>

--047d7bb0498ec035f3050eb2cf7b--