We had originally started with 3 nodes w/ 32GB ram and 768GB SSDs. I pretty much Google'd my way into setting up cassandra and set it up using tokens because I was following an older docco. We were using Cassandra 1.2.5, I learned about vnodes later on and regretted waking up that morning.
1.) I'm not sure if shuffle was successful. We started shuffling on Jun 7th and killed it on the 17th. We let it run over 2 weekends (10 days) and it the node shuffle tool didn't report any meaningful progress. I explained this over IRC and was told `node shuffle` takes a really long time and you shouldn't use it. At the time our ring looked "mostly" balanced so we just killed it. We were migrating from a MongoDB cluster and didn't want to pay for 2 clusters.
2.) During the shuffle we had upped our RF to 2, did not a do a repair and lost 1/3rd of our data. Fortunately we could just use sstable tool to reload the data as it was really deleted.
3.) We ran cleanup a couple days later
4.) Cassandra 1.2.5
After all this, we converted another mongo node we had into Cassandra (same specs) for a cluster of size 4. Now after 4 months, one node (the subject of this thread) is growing faster than the others (which is leading to hot spotting as well). I guess this has to do with the unfinished shuffle? Are there any remedies for this?
On Thursday, September 19, 2013 at 9:50 AM, Robert Coli wrote:
On Wed, Sep 18, 2013 at 4:26 PM, Nimi Wariboko Jr <firstname.lastname@example.org>
When I started with cassandra I had originally set it up to use tokens. I
Are you saying that (other than the imbalance that is the subject of this thread) you were able to use "shuffle" successfully on a cluster with ~150gb per node?
1) How long did it take?
2) Did you experience any difficulties while doing so?
3) Have you run cleanup yet?
4) What version of Cassandra?