Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
From: Michael Fong <michael.fong@ruckuswireless.com>
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: C* 1.2.x vs Gossip marking DOWN/UP
Thread-Topic: C* 1.2.x vs Gossip marking DOWN/UP
Thread-Index: AdGVYYs7FRVB+RbHSxitWyi+FVm8nA==
Date: Wed, 13 Apr 2016 08:58:40 +0000
Message-ID: 
 <CY1PR08MB174078B00DBD7C4F0AC2FB33F5960@CY1PR08MB1740.namprd08.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
spamdiagnosticoutput: 1:23
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative;
	boundary="_000_CY1PR08MB174078B00DBD7C4F0AC2FB33F5960CY1PR08MB1740namp_"
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Apr 2016 08:58:40.1636
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 936dde22-c4d1-470f-b240-618bdde2e61f
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR08MB1740

--_000_CY1PR08MB174078B00DBD7C4F0AC2FB33F5960CY1PR08MB1740namp_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi, all


We have been a Cassandra 4-node cluster (C* 1.2.x) where a node marked all =
the other 3 nodes DOWN, and came back UP a few seconds later. There was a c=
ompaction that kicked in a minute before, roughly 10~MB in size, followed b=
y marking all the other nodes DOWN later. In the other words, in the system=
.log we see
00:00:00 Compacting ....
00:00:03 Compacted 8 sstables ... 10~ megabytes
00:01:06 InetAddress /x.x.x.4 is now DOWN
00:01:06 InetAddress /x.x.x.3 is now DOWN
00:01:06 InetAddress /x.x.x.1 is now DOWN

There was no significant GC activities in gc.log. We have heard that busy c=
ompaction activities would cause this behavior, but we cannot reason why th=
is could happen logically. How come a compaction operation would stop the G=
ossip thread to perform heartbeat check? Has anyone experienced this kind o=
f behavior before?

Thanks in advanced!

Sincerely,

Michael Fong

--_000_CY1PR08MB174078B00DBD7C4F0AC2FB33F5960CY1PR08MB1740namp_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
<meta name=3D"Generator" content=3D"Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
	{font-family:PMingLiU;
	panose-1:2 2 5 0 0 0 0 0 0 0;}
@font-face
	{font-family:PMingLiU;
	panose-1:2 2 5 0 0 0 0 0 0 0;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
	{font-family:"\@PMingLiU";
	panose-1:2 2 5 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:purple;
	text-decoration:underline;}
span.EmailStyle17
	{mso-style-type:personal-compose;
	font-family:"Calibri","sans-serif";
	color:windowtext;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-family:"Calibri","sans-serif";}
@page WordSection1
	{size:8.5in 11.0in;
	margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=3D"EN-US" link=3D"blue" vlink=3D"purple">
<div class=3D"WordSection1">
<p class=3D"MsoNormal">Hi, all<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">We have been a Cassandra 4-node cluster (C* 1.2.x) w=
here a node marked all the other 3 nodes DOWN, and came back UP a few secon=
ds later. There was a compaction that kicked in a minute before, roughly 10=
~MB in size, followed by marking all
 the other nodes DOWN later. In the other words, in the system.log we see<o=
:p></o:p></p>
<p class=3D"MsoNormal">00:00:00 Compacting &#8230;.<o:p></o:p></p>
<p class=3D"MsoNormal">00:00:03 Compacted 8 sstables &#8230; 10~ megabytes<=
o:p></o:p></p>
<p class=3D"MsoNormal">00:01:06 InetAddress /x.x.x.4 is now DOWN<o:p></o:p>=
</p>
<p class=3D"MsoNormal">00:01:06 InetAddress /x.x.x.3 is now DOWN<o:p></o:p>=
</p>
<p class=3D"MsoNormal">00:01:06 InetAddress /x.x.x.1 is now DOWN<o:p></o:p>=
</p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">There was no significant GC activities in gc.log. We=
 have heard that busy compaction activities would cause this behavior, but =
we cannot reason why this could happen logically. How come a compaction ope=
ration would stop the Gossip thread
 to perform heartbeat check? Has anyone experienced this kind of behavior b=
efore?
<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">Thanks in advanced!<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">Sincerely,<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">Michael Fong<o:p></o:p></p>
</div>
</body>
</html>

--_000_CY1PR08MB174078B00DBD7C4F0AC2FB33F5960CY1PR08MB1740namp_--