cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: high i/o usage on one node
Date Mon, 16 Jul 2012 18:27:35 GMT
Is you client balancing between the two nodes ? Heavy writes at CL ONE could result in nodes
dropping messages and having an unbalanced load.

Are you sure there is nothing else running on the machines ? 

Just for fun have you turned off GC logging to see the impact ?

Is there swapping going on ?

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 17/07/2012, at 5:12 AM, feedly team wrote:

> I am having an issue where one node of a 2 node cluster seems to be using much more I/O
than the other node. the cassandra read/write requests seem to be balanced, but iostat shows
the data disk to be maxed at 100% utilization for one machine and <50% for the other. r/s
to be about 3x greater on the high i/o node. I am using a RF of 2 and consistency mode of
ALL for reads and ONE for writes (current requests are very read heavy). user CPU seems to
be fairly low and the same on both machines, but the high i/o machine shows an os load of
34 (!) while the other machine reports 7. I ran a nodetool compactionstats and there are no
tasks pending which i assume means there is no compaction going on, and the logs seem to be
ok as well. the only difference is that on the high i/o node, i am doing full gc logging,
but that's on a separate disk than the data.
> 
> Another oddity is that the high i/o node shows a data size of 86GB while the other shows
71GB. I understand there could be differences, but with a RF of 2 I would think they would
be roughly the equal?
> 
> I am using version 1.0.10.
> 


Mime
View raw message