cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brandon Williams <dri...@gmail.com>
Subject Re: Cassandra benchmark shows OK throughput but high read latency (> 100ms)?
Date Tue, 16 Feb 2010 17:59:26 GMT
On Tue, Feb 16, 2010 at 11:50 AM, Weijun Li <weijunli@gmail.com> wrote:

> Dumped 50mil records into my 2-node cluster overnight, made sure that
> there's not many data files (around 30 only) per Martin's suggestion. The
> size of the data directory is 63GB. Now when I read records from the cluster
> the read latency is still ~44ms, --there's no write happening during the
> read. And iostats shows that the disk (RAID10, 4 250GB 15k SAS) is
> saturated:
>
> Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz
> avgqu-sz   await  svctm  %util
> sda              47.67    67.67 190.33 17.00 23933.33   677.33   118.70
> 5.24   25.25   4.64  96.17
> sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00
> 0.00    0.00   0.00   0.00
> sda2             47.67    67.67 190.33 17.00 23933.33   677.33   118.70
> 5.24   25.25   4.64  96.17
> sda3              0.00     0.00  0.00  0.00     0.00     0.00     0.00
> 0.00    0.00   0.00   0.00
>
> CPU usage is low.
>
> Does this mean disk i/o is the bottleneck for my case? Will it help if I
> increase KCF to cache all sstable index?
>
>
That's exactly what this means.  Disk is slow :(


> Also, this is the almost a read-only mode test, and in reality, our
> write/read ratio is close to 1:1 so I'm guessing read latency will even go
> higher in that case because there will be difficult for cassandra to find a
> good moment to compact the data files that are being busy written.
>

Reads that cause disk seeks are always going to slow things down, since disk
seeks are inherently the slowest operation in a machine.  Writes in
Cassandra should always be fast, as they do not cause any disk seeks.

-Brandon

Mime
View raw message