incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ran Tavory <>
Subject how to decommission two slow nodes?
Date Thu, 20 May 2010 06:57:12 GMT
In my cluster setup I have two datacenters with 5 hosts in one DC and 3 in
the other.
In the 5 hosts DC I'd like to remove two hosts so I'd get 3 and 3 in each.
The two nodes I'd like to decommission have less RAM than the other 3 so
they operate slower.
What's the most effective way to decommission them?

At first I thought I'd decommission the first and then when it's done,
decommission the second, but the problem was that when I decommissioned the
first it started streaming its data to the second node (as well as others I
think) and since the second node was under heavy load, and not enough ram,
it was busy GCing and worked horribly slow. Eventually, after almost 24h of
horribly slow streaming I gave up. This also caused the entire cluster to
operate horribly slow.

So, is there a better way to decommission the two under provisioned nodes
without slowing down the cluster, or at least with a minimum effect?

My replication is 2 and I'm using a RackAwareStrategy so (if everything is
configured correctly with the EndPointSnitch) then at any given time, two
copies of the data exist, one in each DC.


View raw message