cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carlos Alonso <i...@mrcalonso.com>
Subject Re: Read query taking a long time
Date Tue, 20 Oct 2015 17:48:27 GMT
I think also having the output of cfhistograms could help. I'd like to know
how many sstables are being hit during reads.

Also, which CL are you reading with?

cfstats is a local command, so maybe that node you've printed is working
fine but there's another that is causing the latency. Can you check that
command in all nodes?

Regards

Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso>

On 20 October 2015 at 13:59, Brice Figureau <
brice+cassandra@daysofwonder.com> wrote:

> Hi,
>
> Thanks for your answer. Unfortunately since I wrote my e-mail, things
> are a bit better.
>
> This might be because I moved from openjdk 7 to oracle jdk 8 after
> having seen a warning in the C* log about openjdk, and I also added a
> node (for other reasons).
>
> Now the query itself takes only 1.5s~2s instead of the 5s~6s it was
> taking before.
>
> On Mon, 2015-10-19 at 14:38 +0100, Carlos Alonso wrote:
> > Could you send cfhistograms and cfstats relevant to the read column
> > family?
>
> Here are the requested informatrion
> % nodetool proxyhistograms
> proxy histograms
> Percentile      Read Latency     Write Latency     Range Latency
>                     (micros)          (micros)          (micros)
> 50%                  1109.00            372.00           1916.00
> 75%                 14237.00            535.00           2759.00
> 95%                105778.00            642.00           4768.00
> 98%                545791.00            770.00          11864.00
> 99%                785939.00            924.00          14237.00
> Min                    73.00              0.00            373.00
> Max               5839588.00          88148.00          73457.00
>
> % nodetool cfstats akka.messages
> Keyspace: akka
>         Read Count: 3334784
>         Read Latency: 9.98472696792356 ms.
>         Write Count: 7124
>         Write Latency: 0.572256457046603 ms.
>         Pending Flushes: 0
>                 Table: messages
>                 SSTable count: 1
>                 Space used (live): 4680841
>                 Space used (total): 4680841
>                 Space used by snapshots (total): 23615746
>                 Off heap memory used (total): 4051
>                 SSTable Compression Ratio: 0.17318784636027024
>                 Number of keys (estimate): 478
>                 Memtable cell count: 317
>                 Memtable data size: 42293
>                 Memtable off heap memory used: 0
>                 Memtable switch count: 10
>                 Local read count: 3334784
>                 Local read latency: 9.985 ms
>                 Local write count: 7124
>                 Local write latency: 0.573 ms
>                 Pending flushes: 0
>                 Bloom filter false positives: 0
>                 Bloom filter false ratio: 0.00000
>                 Bloom filter space used: 592
>                 Bloom filter off heap memory used: 584
>                 Index summary off heap memory used: 203
>                 Compression metadata off heap memory used: 3264
>                 Compacted partition minimum bytes: 73
>                 Compacted partition maximum bytes: 17436917
>                 Compacted partition mean bytes: 63810
>                 Average live cells per slice (last five minutes):
> 0.6693421039216356
>                 Maximum live cells per slice (last five minutes): 1033.0
>                 Average tombstones per slice (last five minutes): 0.0
>                 Maximum tombstones per slice (last five minutes): 0.0
>
> ----------------
>
> If I read correctly this, there's a huge read latency while proxying,
> but local read latency, or even all node latency on this table is
> correct.
>
> Would that mean this is a network issue?
> --
> Brice Figureau <brice+cassandra@daysofwonder.com>
>
>

Mime
View raw message