cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Walsh, Stephen" <Stephen.Wa...@Aspect.com>
Subject cassandra reads are unbalanced
Date Wed, 02 Dec 2015 11:36:59 GMT
Hey all,

Thanks for taking the time to help.

So we have 6 cassandra nodes in 2 Data Centers.
Both Data Centers have a replication of 3 - so all nodes have all the data.

Over the last 2 days we've noticed that data reads / writes has shifted from balanced to unbalanced
(Nodetool status still shows 100% ownership on every node, with similar sizes)


For Example

We monitor the number of reads / writes of every table via the cassandra JMX metrics. (cassandra.db.read_count)
Over the last hour of this run

Reads
Node 1 (DC1)  =  1.79k    (seeder)
Node 2 (DC1)  =  1.92k
Node 3 (DC1)  =  1.97k

Node 1 (DC2)  =  2.90k   (seeder)
Node 2 (DC2)  =  1.76k
Node 3 (DC2)  =  1.19k

As you see on DC1, everything is pretty well balanced, but on DC2 the reads favour Node1 over
Node 3.
I ran a nodetool repair yesterday - ran for 6 hours and when completed didn't change the read
balance.

Write levels are similar on  DC2, but not as bad a reads.

Anyone any suggestion on how to rebalance? I'm thinking maybe running a nodetool cleanup in
case some of the keys have shifted?

Regards
Stephen Walsh


This email (including any attachments) is proprietary to Aspect Software, Inc. and may contain
information that is confidential. If you have received this message in error, please do not
read, copy or forward this message. Please notify the sender immediately, delete it from your
system and destroy any copies. You may not further disclose or distribute this email or its
attachments.

Mime
View raw message