cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kurt greaves <k...@instaclustr.com>
Subject Re: compaction: huge number of random reads
Date Mon, 07 May 2018 23:12:40 GMT
If you've got small partitions/small reads you should test lowering your
compression chunk size on the table and disabling read ahead. This sounds
like it might just be a case of read amplification.

On Tue., 8 May 2018, 05:43 Kyrylo Lebediev, <Kyrylo_Lebediev@epam.com>
wrote:

> Dear Experts,
>
>
> I'm observing strange behavior on a cluster 2.1.20 during compactions.
>
>
> My setup is:
>
> 12 nodes  m4.2xlarge (8 vCPU, 32G RAM) Ubuntu 16.04, 2T EBS gp2.
>
> Filesystem: XFS, blocksize 4k, device read-ahead - 4k
>
> /sys/block/vxdb/queue/nomerges = 0
>
> SizeTieredCompactionStrategy
>
>
> After data loads when effectively nothing else is talking to the cluster
> and compactions is the only activity, I see something like this:
> $ iostat -dkx 1
> ...
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> xvda              0.00     0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00    0.00    0.00   0.00   0.00
> xvdb              0.00     0.00 4769.00  213.00 19076.00 26820.00
> 18.42     7.95    1.17    1.06    3.76   0.20 100.00
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> xvda              0.00     0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00    0.00    0.00   0.00   0.00
> xvdb              0.00     0.00 6098.00  177.00 24392.00 22076.00
> 14.81     6.46    1.36    0.96   15.16   0.16 100.00
>
> Writes are fine: 177 writes/sec <-> ~22Mbytes/sec,
>
> But for some reason compactions generate a huge number of small reads:
> 6098 reads/s <-> ~24Mbytes/sec.  ===>   Read size is 4k
>
>
> Why instead much smaller amount of large reads I'm getting huge number of
> 4k reads instead?
>
> What could be the reason?
>
> Thanks,
>
> Kyrill
>
>
>

Mime
View raw message