incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: (better info)any way to get the #writes/second, reads per second
Date Tue, 14 May 2013 18:44:55 GMT
> Any reason why cassandra might be reading a lot from the data disks(not
> the commit log disk) more than usual?
On the new node or all nodes ?

Maybe cold Key Cache or cold memmapped files due to a change in the data distribution ?

Did it settle down ? 

Cheers
  

-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 14/05/2013, at 5:06 AM, "Hiller, Dean" <Dean.Hiller@nrel.gov> wrote:

> Ah, okay iostat -x NEEDS a number like "iostat -x 5" works better(first
> one always shows 4% util while second one shows 100%).  Iotop seems a bit
> better here.
> 
> So we know that since we added our new node, we are slammed with read and
> no one is running compations according to "clush -g datanodes nodetool
> compactionstats"
> 
> Any reason why cassandra might be reading a lot from the data disks(not
> the commit log disk) more than usual?
> 
> Thanks,
> Dean
> 
> On 5/13/13 10:46 AM, "Hiller, Dean" <Dean.Hiller@nrel.gov> wrote:
> 
>> We running a pretty consistent load on our cluster and added a new node
>> to a 6 node cluster Friday(QA worked great, but production not so much).
>> One mistake that was made was starting up the new node, then disabling
>> the firewall :( which allowed nodes to discover it BEFORE the node
>> bootstrapped itself.  We shutdown the node and booted him up and he
>> bootstrapped himself streaming all the data in.
>> 
>> After that though, all the ndoes have really really high load numbers
>> now.  We are trying to figure out what is going on still.
>> 
>> Is there any way to get the number of reads/second and writes/second
>> through JMX or something?  The only way I can see of on doing this is
>> manually calculating it by timing the read count and dividing by my
>> manual stop watches start/stop times(timerange).
>> 
>> Also, while my load is load average: 20.31, 19.10, 19.72 , what does a
>> normal iostat look like?  My iostat await time is 13.66 ms which I think
>> is kind of bad, but not that bad to cause a load of 20.31?
>> 
>> Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s
>> avgrq-sz avgqu-sz   await  svctm  %util
>> sda               0.02     0.07   11.70    1.96  1353.67   702.88
>> 150.58     0.19   13.66   3.61   4.93
>> sdb               0.00     0.02    0.11    0.46    20.72    97.54
>> 206.70     0.00    1.33   0.67   0.04
>> 
>> Thanks,
>> Dean
> 


Mime
View raw message