cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Weijun Li <weiju...@gmail.com>
Subject Unbalanced read latency among nodes in a cluster
Date Fri, 19 Feb 2010 20:03:37 GMT
I setup a two cassandra clusters with 2 nodes each. Both use random
partitioner. It's strange that for each cluster, one node has much shortter
read latency than the other one

This is the info of one of the cluster:

Node A: read count 77302, data file 41GB, read latency 58180, io saturation
100%
Node B: read count 488753, data file 26GB, read latency 5822 , io saturation
35%.

I first started node A, then ran B to join the cluster. Both machines have
exactly the same hardware and OS. The test client randomly pick a node to
write and it worked fine for the other cluster.

Address       Status     Load
Range                                      Ring

169400792707028208569145873749456918214
10.xxxxxxx Up         38.39 GB
103633195217832666843316719920043079797    |<--|
10.xxxxxxx Up         24.22 GB
169400792707028208569145873749456918214    |-->|

For both clusters, whichever node that took more reads (with larger data
file) owns the much worse read latency.

What's the algorithm that cassandra use to split token when a new node is
joining? What could cause this unbalanced read latency issue? How can I fix
this? How to make sure all nodes get evenly distributed data and traffic?

-Weijun

Mime
View raw message