Do you have three rows (or a small number) that are written to, or read from more frequently than the others?

Remember your data is distributed based on the key, so if you have a very active index on a particular, or small set of, row(s) you could see strange modalities like this.

Ben

On Thu, Jul 12, 2012 at 2:31 AM, Olivier Mallassi <omallassi@octo.com> wrote:
yep, all nodes have a similar token range ownership and we use randompartitionner. 

After the initial load, all my nodes have the same amount of data (aroud 2Go each)
In fact, my guesses is that these load tests were only using a small part of our data set (and that data was routed to this couple of nodes and replicas)
Running other load tests that imply the complete data set, I do not see the same symptoms. Local Read/Writes are much more balanced across all nodes. 

Anyway, if further tests confirmed my data is unbalanced (or not very well balanced), I guess my only way to resolve this is to change the token assignment to split more the data on nodes that are the most used. Am i wrong? 



On Wed, Jul 11, 2012 at 5:39 PM, rohit bhatia <rohit2412@gmail.com> wrote:
The requests are clearly balanced.. But the data is not

Do all nodes have the similar token range ownership?
Are you using a randomPartitioner or an OrderPreserving partitioner? 


On Wed, Jul 11, 2012 at 7:29 PM, Olivier Mallassi <omallassi@octo.com> wrote:
Hi all

During our load tests, we observe that a couple of servers are much more used than others (in terms of CPU). In front of the cassandra cluster we have a IPVS (ipvsadm configured in rr)

in opscenter we have the following graph : 
Write / read Request : almost evenly distributed ...
Inline image 1

Local Writes / Read Request : clearly not evenly distributed and as you can imagine, the high loaded nodes (in CPU) are the ones with more TPS....
Inline image 2


Based on that, my best idea is that load tests we are doing use data that is physically located on a couple of server (bad luck....). 

We tried using different data (in our load tests), so different key but the behavior seem identical (we will repeat the tests). 

Do you have any ideas of what can be wrong? 

thx a lot. 

--
............................................................
Olivier Mallassi
OCTO Technology
............................................................
50, Avenue des Champs-Elysées
75008 Paris

Mobile: (33) 6 28 70 26 61
Tél: (33) 1 58 56 10 00
Fax: (33) 1 58 56 10 01

http://www.octo.com
Octo Talks! http://blog.octo.com






--
............................................................
Olivier Mallassi
OCTO Technology
............................................................
50, Avenue des Champs-Elysées
75008 Paris

Mobile: (33) 6 28 70 26 61
Tél: (33) 1 58 56 10 00
Fax: (33) 1 58 56 10 01

http://www.octo.com
Octo Talks! http://blog.octo.com





--
Ben Coverston
DataStax -- The Apache Cassandra Company