cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Laing, Michael" <michael.la...@nytimes.com>
Subject Re: Read query taking a long time
Date Wed, 21 Oct 2015 11:47:41 GMT
Are the clocks synchronized across the cluster - probably, but I thought I
would ask :)

On Wed, Oct 21, 2015 at 3:35 AM, Brice Figureau <
brice+cassandra@daysofwonder.com> wrote:

> Hi,
>
> On 20/10/2015 19:48, Carlos Alonso wrote:
> > I think also having the output of cfhistograms could help. I'd like to
> > know how many sstables are being hit during reads.
>
> Ooops, my bad I thought you mentioned proxyhistograms.
> Here's the worst node cfhistograms:
>
> nodetool cfhistograms akka messages
> akka/messages histograms
> Percentile  SSTables     Write Latency      Read Latency    Partition
> Size        Cell Count
>                               (micros)          (micros)           (bytes)
> 50%             1.00             60.00            310.00
>  535                 1
> 75%             2.00            103.00           9887.00
>  11864                17
> 95%             3.00          14237.00         126934.00
>  73457               215
> 98%             3.00          51012.00         219342.00
> 263210               446
> 99%             3.00          61214.00        1131752.00
> 454826               924
> Min             0.00              6.00             18.00
> 73                 0
> Max             6.00         263210.00        5839588.00
> 10090808              1109
>
>
> > Also, which CL are you reading with?
>
> For the test request in cqlsh, I'm using ONE, but the application in which
> I first observed the issue was using QUORUM.
>
> > cfstats is a local command, so maybe that node you've printed is working
> > fine but there's another that is causing the latency. Can you check that
> > command in all nodes?
>
> I checked and on all nodes, the read latency and read local latency are
> within 15 to 40ms.
>
> I also noticed that C* was taking a fair bit of CPU on some of the nodes
> (ranging from 60% to 200%), looking at ttop output it was mostly taken by
> SharedPool-Worker threads, which I assume are the thread that are doing the
> real query work.
>
> Well, I'm puzzled, and I'll keep searching, thanks for your help!
> --
> Brice Figureau
>

Mime
View raw message