I am running some tests trying to scale out our application from using a 3 node cluster to 6 node cluster. The thing I observed is that when using a 3 node cluster I was able to handle abt 41 req/second, so I added 3 more nodes thinking it should close to double, but instead it only goes upto bat 47 req/second!! I am doing something wrong and it is not obvious, so wanted some help in what stats could/should I monitor to tell me things like if a node has more requests or if the load distribution is not random enough?
Note I am using direct thrift (old code base) and cassandra 1.1.6. The data model is for storing blobs (split across columns) and has around 6 CF, RF=3 and all operations are at quorum. Also at the end of the run nodetool ring reports the same data size.