cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Unbalanced read latency among nodes in a cluster
Date Fri, 19 Feb 2010 21:04:14 GMT
http://wiki.apache.org/cassandra/Operations

On Fri, Feb 19, 2010 at 3:03 PM, Weijun Li <weijunli@gmail.com> wrote:
> I setup a two cassandra clusters with 2 nodes each. Both use random
> partitioner. It's strange that for each cluster, one node has much shortter
> read latency than the other one
>
> This is the info of one of the cluster:
>
> Node A: read count 77302, data file 41GB, read latency 58180, io saturation
> 100%
> Node B: read count 488753, data file 26GB, read latency 5822 , io saturation
> 35%.
>
> I first started node A, then ran B to join the cluster. Both machines have
> exactly the same hardware and OS. The test client randomly pick a node to
> write and it worked fine for the other cluster.
>
> Address       Status     Load
> Range                                      Ring
>
> 169400792707028208569145873749456918214
> 10.xxxxxxx Up         38.39 GB
> 103633195217832666843316719920043079797    |<--|
> 10.xxxxxxx Up         24.22 GB
> 169400792707028208569145873749456918214    |-->|
>
> For both clusters, whichever node that took more reads (with larger data
> file) owns the much worse read latency.
>
> What's the algorithm that cassandra use to split token when a new node is
> joining? What could cause this unbalanced read latency issue? How can I fix
> this? How to make sure all nodes get evenly distributed data and traffic?
>
> -Weijun
>

Mime
View raw message