Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
From: Anubhav Kale <Anubhav.Kale@microsoft.com>
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
CC: Sean Usher <seusher@exchange.microsoft.com>
Subject: RemoveNode CPU Spike Question
Thread-Topic: RemoveNode CPU Spike Question
Thread-Index: AdJrawShXAXCMTC5RMCBa9RE/z4HxA==
Date: Tue, 10 Jan 2017 17:57:37 +0000
Message-ID: <CY1PR00MB0124BD90A98D38841141633D8B670@CY1PR00MB0124.namprd00.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
received-spf: None (protection.outlook.com: microsoft.com does not designate
 permitted sender hosts)
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative;
	boundary="_000_CY1PR00MB0124BD90A98D38841141633D8B670CY1PR00MB0124namp_"
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Jan 2017 17:57:37.8752
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR00MB0156
archived-at: Tue, 10 Jan 2017 17:57:51 -0000

--_000_CY1PR00MB0124BD90A98D38841141633D8B670CY1PR00MB0124namp_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hello,

Recently, I started noticing an interesting pattern. When I execute "remove=
node", a subset of the nodes that now own the tokens result it in a CPU spi=
ke / disk activity, and sometimes SSTables on those nodes shoot up.

After looking through the code, it appears to me that below function forces=
 data to be streamed from some of the new nodes to the node from where "rem=
ovenode" is kicked in. Is my understanding correct ?

https://github.com/apache/cassandra/blob/d384e781d6f7c028dbe88cfe9dd3e966e7=
2cd046/src/java/org/apache/cassandra/service/StorageService.java#L2548

Our nodes don't run very hot, but it appears this streaming causes them to =
have issues. Have other people seen this ?

Thanks !

--_000_CY1PR00MB0124BD90A98D38841141633D8B670CY1PR00MB0124namp_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:#0563C1;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:#954F72;
	text-decoration:underline;}
span.EmailStyle17
	{mso-style-type:personal-compose;
	font-family:"Calibri",sans-serif;
	color:windowtext;}
.MsoChpDefault
	{mso-style-type:export-only;}
@page WordSection1
	{size:8.5in 11.0in;
	margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=3D"EN-US" link=3D"#0563C1" vlink=3D"#954F72">
<div class=3D"WordSection1">
<p class=3D"MsoNormal">Hello,<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">Recently, I started noticing an interesting pattern.=
 When I execute &#8220;removenode&#8221;, a subset of the nodes that now ow=
n the tokens result it in a CPU spike / disk activity, and sometimes SSTabl=
es on those nodes shoot up.<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">After looking through the code, it appears to me tha=
t below function forces data to be streamed from some of the new nodes to t=
he node from where &#8220;removenode&#8221; is kicked in. Is my understandi=
ng correct ?
<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal"><a href=3D"https://github.com/apache/cassandra/blob/=
d384e781d6f7c028dbe88cfe9dd3e966e72cd046/src/java/org/apache/cassandra/serv=
ice/StorageService.java#L2548">https://github.com/apache/cassandra/blob/d38=
4e781d6f7c028dbe88cfe9dd3e966e72cd046/src/java/org/apache/cassandra/service=
/StorageService.java#L2548</a><o:p></o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">Our nodes don&#8217;t run very hot, but it appears t=
his streaming causes them to have issues. Have other people seen this ?
<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">Thanks !<o:p></o:p></p>
</div>
</body>
</html>

--_000_CY1PR00MB0124BD90A98D38841141633D8B670CY1PR00MB0124namp_--