On Tue, Feb 16, 2010 at 11:50 AM, Weijun Li <weijunli@gmail.com> wrote:
Dumped 50mil records into my 2-node cluster overnight, made sure that there's not many data files (around 30 only) per Martin's suggestion. The size of the data directory is 63GB. Now when I read records from the cluster the read latency is still ~44ms, --there's no write happening during the read. And iostats shows that the disk (RAID10, 4 250GB 15k SAS) is saturated:

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda              47.67    67.67 190.33 17.00 23933.33   677.33   118.70     5.24   25.25   4.64  96.17
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2             47.67    67.67 190.33 17.00 23933.33   677.33   118.70     5.24   25.25   4.64  96.17
sda3              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

CPU usage is low.

Does this mean disk i/o is the bottleneck for my case? Will it help if I increase KCF to cache all sstable index?


That's exactly what this means.  Disk is slow :(
 
Also, this is the almost a read-only mode test, and in reality, our write/read ratio is close to 1:1 so I'm guessing read latency will even go higher in that case because there will be difficult for cassandra to find a good moment to compact the data files that are being busy written.

Reads that cause disk seeks are always going to slow things down, since disk seeks are inherently the slowest operation in a machine.  Writes in Cassandra should always be fast, as they do not cause any disk seeks.

-Brandon