cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Dabbs" <dmda...@gmail.com>
Subject Tracking down read latency
Date Thu, 03 Feb 2011 05:35:24 GMT
Hello.

We’re encountering some high read latency issues. But our main Cass expert
is out-of-office so it falls to me.
We're more read than write, though there doesn't seem to be many pending
reads.
I have seen active/pending row-read at three or four, though.

Pool Name                    Active   Pending      Completed
FILEUTILS-DELETE-POOL             0         0             46
STREAM-STAGE                      0         0              0
RESPONSE-STAGE                    0         0       17471880
ROW-READ-STAGE                    1         1       37652361
LB-OPERATIONS                     0         0              0
MISCELLANEOUS-POOL                0         0              0
GMFD                              0         0         154630
LB-TARGET                         0         0              0
CONSISTENCY-MANAGER               0         0        2993464
ROW-MUTATION-STAGE                0         0       16383305
MESSAGE-STREAMING-POOL            0         0              0
LOAD-BALANCER-STAGE               0         0              0
FLUSH-SORTER-POOL                 0         0              0
MEMTABLE-POST-FLUSHER             0         0            116
FLUSH-WRITER-POOL                 0         0            116
AE-SERVICE-STAGE                  0         0              0
HINTED-HANDOFF-POOL               0         0             16


Does the high iops on our data mean we need to tune Key or other caches?
 
$ iostat
Linux 2.6.18-194.11.3.el5 02/03/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.00    0.00    0.25    1.32    0.00   97.43

Device:            tps   Blk_read/s   Blk_wrtn/s     Blk_read    Blk_wrtn
sda               1.52         6.66        20.53     88886262
  273904650
sda1              0.00         0.00         0.00         2484
         18
sda2              1.46         5.68        18.95     75741946
  252831120
sda3              0.06         0.98         1.58     13141400
   21073512
# data here
sdb             103.06     13964.72      2718.28 186315436859 36266884800
sdb1            103.06     13964.72      2718.28 186315436235 36266884800
# commit logs here
sdc               1.47         1.71       309.36     22800725
 4127423000
sdc1              1.47         1.71       309.36     22799901
 4127423000



We're running on a beefy 64-bit Nehalem, so mmap should be
available/possible.
I need to check with our Cassandra lead when he's available as to why we're
not using mmap or auto.

>From /opt/cassandra/conf/storage-conf.xml. 

    <DiskAccessMode>standard</DiskAccessMode>



Heap size is 16gb.

JVM_OPTS=" \
        -ea \
        -Xms16G \
        -Xmx16G \
        -XX:+UseParNewGC \
        -XX:+UseConcMarkSweepGC \
        -XX:+CMSParallelRemarkEnabled \
        -XX:SurvivorRatio=8 \
        -XX:MaxTenuringThreshold=1 \
        -XX:CMSInitiatingOccupancyFraction=75 \
        -XX:+UseCMSInitiatingOccupancyOnly \
        -XX:+HeapDumpOnOutOfMemoryError \
        -XX:+UseCompressedOops \
        -XX:+UseThreadPriorities \
        -XX:ThreadPriorityPolicy=42 \
        -Dcassandra.compaction.priority=1"




If I've omitted any key infos, please advise and I'll provide.


Thanks,

David




Mime
View raw message