cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Schuller <peter.schul...@infidyne.com>
Subject Re: what causes MESSAGE-DESERIALIZER-POOL to spike
Date Tue, 27 Jul 2010 07:23:53 GMT
> Ah, the weird thing is I/O is assumed to be the limiting factor, but iops on
> the box was very low. Service time and atime very low, and the data access
> was only 6MB a second. With all of this, I'm tending to believe that the
> problem may be someplace else.

You vmstat output shows idle and wait time. I suspect an "iostat -x 1"
will show that your not keeping your underlying device busy (the
right-most column not being stuck at 100% or close to it). Is this the
case? If it is close to 100% or at 100% you'd want to look at the
average queue size column too. But given the vmstat output I doubt
this is the case since you should either be seeing a lot more wait
time or a lot less idle time.

The question is what the limiting factor is. What does jconsole/etc
say about the state of the threads in ROW-READ-STAGE? Statistically if
you poll them a few times; what does it mostly seem to be waiting on?
Normally, if the expectation is that ROW-READ-STAGE is disk bound,
that should show up by the threads usually being busy waiting for disk
I/O upon repeated polling (or waiting for more work to do if they are
idle).

(And btw for some reason I totally missed the fact that ROW-READ-STAGE
had 4096 pending and 8 active... oh well.)

-- 
/ Peter Schuller

Mime
View raw message