cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Walsh, Stephen" <Stephen.Wa...@Aspect.com>
Subject RE: cassandra reads are unbalanced
Date Fri, 04 Dec 2015 15:51:16 GMT
Thanks for your input, but I think I’ve already answered most of your questions.


How many clients do you have performing reads?

------------------
On Wed, Dec 2, 2015 at 6:44 PM, Walsh, Stephen <Stephen.Walsh@aspect.com<mailto:Stephen.Walsh@aspect.com>>
wrote
….
There are 2 application (1 for each DC) who read and write at the same rate to their local
DC
….
--------------------






Is your load balancer in front of your clients or between your clients and Cassandra?

------------------
On Thu, Dec 3, 2015 at 4:58 AM, Walsh, Stephen <Stephen.Walsh@aspect.com<mailto:Stephen.Walsh@aspect.com>>
wrote:
…
our production applications are behind a round robin load balancer
…
------------------

No Load Balancers talk to cassandra – I’m only mentioning this to show that the writes
/ read are evenly distributed over the 2 DC’s






Does Node1 of DC2 have the exact same configuration of hardware of the other nodes
Yes





Is it in the same rack
It’s in AWS – but we have it configured via the GossipProperytFileSnitch that they are
all on unique racks






Maybe your load balancer thinks that node is more capable and handles requests faster so that
it looks less loaded than the other two nodes
Unlikely, it’s all TCP SSL pass though connections. It doesn’t balance on load, it just
round robins each request





You might also check the read counts after a very short interval of time to see if Node1 is
uniformly getting more requests or just occasionally
------------------
On Wed, Dec 2, 2015 at 3:36 PM, Walsh, Stephen <Stephen.Walsh@aspect.com<mailto:Stephen.Walsh@aspect.com>>
wrote …
We monitor the number of reads / writes of every table via the cassandra JMX metrics. (cassandra.db.read_count)
…
------------------
We can only monitor in 1 hour moving window




Maybe the other two nodes are in a different rack that occasionally has net connectivity issues
Unlikely seems its AWS






From: Jack Krupansky [mailto:jack.krupansky@gmail.com]
Sent: 03 December 2015 16:11
To: user@cassandra.apache.org
Subject: Re: cassandra reads are unbalanced

How many clients do you have performing reads?

Is your load balancer in front of your clients or between your clients and Cassandra?

Does Node1 of DC2 have the exact same configuration of hardware of the other nodes? Is it
in the same rack? Maybe your load balancer thinks that node is more capable and handles requests
faster so that it looks less loaded than the other two nodes.

You might also check the read counts after a very short interval of time to see if Node1 is
uniformly getting more requests or just occasionally. Maybe the other two nodes are in a different
rack that occasionally has net connectivity issues so that the requests get diverted by the
client/load balancer to Node1 during those times.


-- Jack Krupansky

On Thu, Dec 3, 2015 at 4:58 AM, Walsh, Stephen <Stephen.Walsh@aspect.com<mailto:Stephen.Walsh@aspect.com>>
wrote:
Thanks but keep in mind that both DC should be getting the same load, our production applications
are behind a round robin load balancer – so each one our local application talk to its local
Cassandra DataCenter.

It took about 4 hours but the nodetool cleanup eventually balanced all nodes

From: DuyHai Doan [mailto:doanduyhai@gmail.com<mailto:doanduyhai@gmail.com>]
Sent: 02 December 2015 16:27

To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: cassandra reads are unbalanced

If you're using the Java driver with LOCAL_ONE and the default load balancing strategy (TokenAware
wrapped on DCAwareRoundRobin), the driver will always select the primary replica. To change
this behavior and introduce some randomness so that non primary replicas get a chance to serve
a read:

new TokenAwarePolicy(new DCAwareRoundRobinPolicy("local_DC"), true).

The second parameter (true) asks the TokenAware policy to "shuffle" replica on each request
to avoid always returning the primary replica.

On Wed, Dec 2, 2015 at 6:44 PM, Walsh, Stephen <Stephen.Walsh@aspect.com<mailto:Stephen.Walsh@aspect.com>>
wrote:
Very good questions.

We have reads and writes at LOCAL_ONE.
There are 2 application (1 for each DC) who read and write at the same rate to their local
DC
(All reads / writes started all perfectly even and degraded over time)

We use DCAwareRoundRobin policy

On update on the nodetool cleanup – it has help but hasn’t balanced all nodes. Node 1
on DC2 is still quite high

Node 1 (DC1)  =  1.35k    (seeder)
Node 2 (DC1)  =  1.54k
Node 3 (DC1)  =  1.45k

Node 1 (DC2)  =  2.06k   (seeder)
Node 2 (DC2)  =  1.38k
Node 3 (DC2)  =  1.43k


From: DuyHai Doan [mailto:doanduyhai@gmail.com<mailto:doanduyhai@gmail.com>]
Sent: 02 December 2015 14:22
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: cassandra reads are unbalanced

Which Consistency level do you use for reads ? ONE ? Are you reading from only DC1 or from
both DC ?
What is the LoadBalancingStrategy you have configured for your driver ? TokenAware wrapped
on DCAwareRoundRobin ?





On Wed, Dec 2, 2015 at 3:36 PM, Walsh, Stephen <Stephen.Walsh@aspect.com<mailto:Stephen.Walsh@aspect.com>>
wrote:
Hey all,

Thanks for taking the time to help.

So we have 6 cassandra nodes in 2 Data Centers.
Both Data Centers have a replication of 3 – so all nodes have all the data.

Over the last 2 days we’ve noticed that data reads / writes has shifted from balanced to
unbalanced
(Nodetool status still shows 100% ownership on every node, with similar sizes)


For Example

We monitor the number of reads / writes of every table via the cassandra JMX metrics. (cassandra.db.read_count)
Over the last hour of this run

Reads
Node 1 (DC1)  =  1.79k    (seeder)
Node 2 (DC1)  =  1.92k
Node 3 (DC1)  =  1.97k

Node 1 (DC2)  =  2.90k   (seeder)
Node 2 (DC2)  =  1.76k
Node 3 (DC2)  =  1.19k

As you see on DC1, everything is pretty well balanced, but on DC2 the reads favour Node1 over
Node 3.
I ran a nodetool repair yesterday – ran for 6 hours and when completed didn’t change the
read balance.

Write levels are similar on  DC2, but not as bad a reads.

Anyone any suggestion on how to rebalance? I’m thinking maybe running a nodetool cleanup
in case some of the keys have shifted?

Regards
Stephen Walsh


This email (including any attachments) is proprietary to Aspect Software, Inc. and may contain
information that is confidential. If you have received this message in error, please do not
read, copy or forward this message. Please notify the sender immediately, delete it from your
system and destroy any copies. You may not further disclose or distribute this email or its
attachments.

This email (including any attachments) is proprietary to Aspect Software, Inc. and may contain
information that is confidential. If you have received this message in error, please do not
read, copy or forward this message. Please notify the sender immediately, delete it from your
system and destroy any copies. You may not further disclose or distribute this email or its
attachments.

This email (including any attachments) is proprietary to Aspect Software, Inc. and may contain
information that is confidential. If you have received this message in error, please do not
read, copy or forward this message. Please notify the sender immediately, delete it from your
system and destroy any copies. You may not further disclose or distribute this email or its
attachments.

This email (including any attachments) is proprietary to Aspect Software, Inc. and may contain
information that is confidential. If you have received this message in error, please do not
read, copy or forward this message. Please notify the sender immediately, delete it from your
system and destroy any copies. You may not further disclose or distribute this email or its
attachments.
Mime
View raw message