cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Dabbs" <dmda...@gmail.com>
Subject RE: Tracking down read latency
Date Fri, 04 Feb 2011 19:44:01 GMT

Thank you both for your advice. See my updated iostats below.


>From: sridhar.basam@gmail.com [mailto:sridhar.basam@gmail.com] On Behalf Of
sridhar basam
>Sent: Thursday, February 03, 2011 10:58 AM
>To: user@cassandra.apache.org
>Subject: Re: Tracking down read latency
>
>The data provided is also a average value since boot time. Run the -x as
suggested below but run it via a interval of around 5 seconds. You very well
could be having i/o issue, >it is hard to tell from the overall average
value you provided. Collect "iostat -x 5" during the times when you see slow
reads and see how busy the disks are.

>Sridhar

>On Thu, Feb 3, 2011 at 3:21 AM, Peter Schuller wrote:
>> $ iostat
>
>As rcoli already mentioned you don't seen to have an I/O problem, but
>as a point of general recommendation: When determining whether you are
>blocking on disk I/O, pretty much *always* use "iostat -x" rather than
>the much less useful default mode of iostat. The %util and queue
>wait/average time columns are massively useful/important; without them
>one is much more blind as to whether or not storage devices are
>actually saturated.
>
> Peter Schuller


Our data is on sdb, commit logs on sdc.
So do I read this correctly that we're 'await'ing 6+millis on average for
data drive (sdb) 
requests to be serviced?


$iostat -x 5

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.59    0.00    0.22    0.94    0.00   98.25

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz
avgqu-sz   await  svctm  %util
sda               0.00     0.00  0.00  0.00     0.00     0.00     0.00
0.00    0.00   0.00   0.00
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
0.00    0.00   0.00   0.00
sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00
0.00    0.00   0.00   0.00
sda3              0.00     0.00  0.00  0.00     0.00     0.00     0.00
0.00    0.00   0.00   0.00
sdb              11.20     0.00 42.00  0.00  4993.60     0.00   118.90
0.28    6.77   5.22  21.92
sdb1             11.20     0.00 42.00  0.00  4993.60     0.00   118.90
0.28    6.77   5.22  21.92
sdc               0.00    31.00  0.00  1.40     0.00   259.20   185.14
0.00    0.14   0.14   0.02
sdc1              0.00    31.00  0.00  1.40     0.00   259.20   185.14
0.00    0.14   0.14   0.02

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.56    0.00    0.18    1.08    0.00   98.17

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz
avgqu-sz   await  svctm  %util
sda               0.00     0.00  0.00  0.00     0.00     0.00     0.00
0.00    0.00   0.00   0.00
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
0.00    0.00   0.00   0.00
sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00
0.00    0.00   0.00   0.00
sda3              0.00     0.00  0.00  0.00     0.00     0.00     0.00
0.00    0.00   0.00   0.00
sdb               8.80     0.00 49.40  0.00  5936.00     0.00   120.16
0.33    6.62   5.22  25.78
sdb1              8.80     0.00 49.40  0.00  5936.00     0.00   120.16
0.33    6.62   5.22  25.78
sdc               0.00     0.00  0.00  0.00     0.00     0.00     0.00
0.00    0.00   0.00   0.00
sdc1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.99    0.00    0.22    1.08    0.00   97.71

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz
avgqu-sz   await  svctm  %util
sda               0.00     0.00  0.00  0.00     0.00     0.00     0.00
0.00    0.00   0.00   0.00
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
0.00    0.00   0.00   0.00
sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00
0.00    0.00   0.00   0.00
sda3              0.00     0.00  0.00  0.00     0.00     0.00     0.00
0.00    0.00   0.00   0.00
sdb              11.40     0.00 46.20  0.00  5147.20     0.00   111.41
0.30    6.55   5.58  25.80
sdb1             11.40     0.00 46.20  0.00  5147.20     0.00   111.41
0.30    6.55   5.58  25.80
sdc               0.00     7.40  0.00  0.80     0.00    65.60    82.00
0.00    0.25   0.25   0.02
sdc1              0.00     7.40  0.00  0.80     0.00    65.60    82.00
0.00    0.25   0.25   0.02

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.68    0.00    0.23    0.95    0.00   98.13

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz
avgqu-sz   await  svctm  %util
sda               0.00     0.80  0.00  0.80     0.00    12.77    16.00
0.00    0.25   0.25   0.02
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
0.00    0.00   0.00   0.00
sda2              0.00     0.80  0.00  0.80     0.00    12.77    16.00
0.00    0.25   0.25   0.02
sda3              0.00     0.00  0.00  0.00     0.00     0.00     0.00
0.00    0.00   0.00   0.00
sdb               5.19     0.00 38.12  0.00  4356.09     0.00   114.26
0.26    6.70   5.91  22.53
sdb1              5.19     0.00 38.12  0.00  4356.09     0.00   114.26
0.26    6.70   5.91  22.53
sdc               0.00     0.00  0.00  0.00     0.00     0.00     0.00
0.00    0.00   0.00   0.00
sdc1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
0.00    0.00   0.00   0.00




Mime
View raw message