cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anubhav Kale <Anubhav.K...@microsoft.com>
Subject RE: RemoveNode CPU Spike Question
Date Tue, 10 Jan 2017 18:03:12 GMT
Well, looking through logs I confirmed that my understanding below is correct, but would be
good to hear from experts for sure 😊

From: Anubhav Kale [mailto:Anubhav.Kale@microsoft.com]
Sent: Tuesday, January 10, 2017 9:58 AM
To: user@cassandra.apache.org
Cc: Sean Usher <seusher@exchange.microsoft.com>
Subject: RemoveNode CPU Spike Question

Hello,

Recently, I started noticing an interesting pattern. When I execute “removenode”, a subset
of the nodes that now own the tokens result it in a CPU spike / disk activity, and sometimes
SSTables on those nodes shoot up.

After looking through the code, it appears to me that below function forces data to be streamed
from some of the new nodes to the node from where “removenode” is kicked in. Is my understanding
correct ?

https://github.com/apache/cassandra/blob/d384e781d6f7c028dbe88cfe9dd3e966e72cd046/src/java/org/apache/cassandra/service/StorageService.java#L2548<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fcassandra%2Fblob%2Fd384e781d6f7c028dbe88cfe9dd3e966e72cd046%2Fsrc%2Fjava%2Forg%2Fapache%2Fcassandra%2Fservice%2FStorageService.java%23L2548&data=02%7C01%7CAnubhav.Kale%40microsoft.com%7C173daa48fcaf4ca6498d08d43982318c%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636196678720784947&sdata=JZ9zWh%2FtJJ%2FbhXXkT41yQhANKaUSBHfP53WraY2vL8M%3D&reserved=0>

Our nodes don’t run very hot, but it appears this streaming causes them to have issues.
Have other people seen this ?

Thanks !
Mime
View raw message