incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nimi Wariboko Jr <nimiwaribo...@gmail.com>
Subject Re: Rebalancing vnodes cluster
Date Thu, 19 Sep 2013 23:45:19 GMT
We had originally started with 3 nodes w/ 32GB ram and 768GB SSDs. I pretty much Google'd my
way into setting up cassandra and set it up using tokens because I was following an older
docco. We were using Cassandra 1.2.5, I learned about vnodes later on and regretted waking
up that morning.

1.) I'm not sure if shuffle was successful. We started shuffling on Jun 7th and killed it
on the 17th. We let it run over 2 weekends (10 days) and it the node shuffle tool didn't report
any meaningful progress. I explained this over IRC and was told `node shuffle` takes a really
long time and you shouldn't use it. At the time our ring looked "mostly" balanced so we just
killed it. We were migrating from a MongoDB cluster and didn't want to pay for 2 clusters.
2.) During the shuffle we had upped our RF to 2, did not a do a repair and lost 1/3rd of our
data. Fortunately we could just use sstable tool to reload the data as it was really deleted.
3.) We ran cleanup a couple days later
4.) Cassandra 1.2.5

After all this, we converted another mongo node we had into Cassandra (same specs) for a cluster
of size 4. Now after 4 months, one node (the subject of this thread) is growing faster than
the others (which is leading to hot spotting as well). I guess this has to do with the unfinished
shuffle? Are there any remedies for this? 

On Thursday, September 19, 2013 at 9:50 AM, Robert Coli wrote:

> On Wed, Sep 18, 2013 at 4:26 PM, Nimi Wariboko Jr <nimiwaribokoj@gmail.com (mailto:nimiwaribokoj@gmail.com)>
wrote:
> > When I started with cassandra I had originally set it up to use tokens. I
> > then migrated to vnodes (using shuffle), but my cluster isn't balanced (http://imgur.com/73eNhJ3).

> 
> Are you saying that (other than the imbalance that is the subject of this thread) you
were able to use "shuffle" successfully on a cluster with ~150gb per node?
> 
> 1) How long did it take?
> 2) Did you experience any difficulties while doing so?
> 3) Have you run cleanup yet?
> 4) What version of Cassandra?
> 
> =Rob
>  
> 
> 
> 
> 
> 



Mime
View raw message