cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alain RODRIGUEZ <arodr...@gmail.com>
Subject Re: Upgredesstables doing 4K reads
Date Thu, 23 Jun 2016 07:48:19 GMT
Hi,

Sorry no one get back to you yet. Do you still have the issue?

It's unclear to me what produces this yet. A few ideas though:

We are quite pedantic about OS settings. All nodes got same settings
> and C* configuration.


Considering this hypothesis, I hope that's 100% true.

2 nodes behaving badly out of 6, makes me think of an unbalanced cluster.
Do you use RF=2 there ? do you have wide rows or unbalanced data (partition
keys not well distributes)?

Could you check and paste the output from nodetool cfstats and nodetool
cfhistograms on the most impacting tables ?

Could those nodes have hardware issues of some kind ?

C*heers,
-----------------------
Alain Rodriguez - alain@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com



2016-06-02 13:43 GMT+02:00 Jacek Luczak <difrost.kernel@gmail.com>:

> Hi,
>
> I've got a 6 node C* cluster (all nodes are equal both in OS and HW
> setup, they are DL380 Gen9 with Smart Array RAID 50,3 on SAS 15K HDDs)
> which has been recently updated from 2.2.5 to 3.5. As part of the
> update I've done the upgradesstables.
>
> On 4 nodes the average request size issued to the block dev was never
> higher than 8 (that maps to 4K reads) while on remaining 2 nodes it
> was basically always maxed 512 (256K reads).
>
> Nodes doing 4K reads were pumping max 2K read IOPs while the 2 nodes
> never went up above 30 IOPs.
>
> We are quite pedantic about OS settings. All nodes got same settings
> and C* configuration. On all nodes block dev got noop scheduler set
> and read ahead aligned with strip size.
>
> During heavy read workloads we've also noticed that those 4 nodes can
> swing up to 10K IOPs to get data from storage, the 2 are much below.
>
> What can cause such difference?
>
> -Jacek
>

Mime
View raw message