cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lucas Benevides <lu...@maurobenevides.com.br>
Subject Re: Why Would a Single Node Drive Read Latency 5x Worse & Shutting it Down Improves it
Date Fri, 09 Mar 2018 18:06:33 GMT
Hi Habash,

If there is not data being written it is really a mistery.
What is the Consistency Level used in this test and what Replication Factor?

Independently from the above answers, you should take a look in the
following metrics: CoordinatorReadLatency and CoordinatorWriteLatency. And
you compare with your latency.

Best regards,
Lucas Benevides Dias

2018-03-09 11:46 GMT-03:00 Fd Habash <fmhabash@gmail.com>:

>
>
> Tying to come up with an explanation for this real-life C* behavior. Under
> a constant load test of 2k reads/sec, the very top graph is the read
> latency for a single node which is a seed one.
>
>
>
> You may not see it here, but there are 8 other nodes that follow it. Since
> we have 4 nodes in each of the 3 AZ’s, I know these other 8 nodes (4 per
> AZ) hold the secondary replicas for the seed node.
>
>
>
> At the time of taking this metric snapshot, there were 0 compactions, 0
> anti-compactions, 0 repairs. Cluster did nothing else but serve read
> requests.
>
>
>
> Shortly after the 0900, I stopped C* on the top seed node and read latency
> dropped from 1 sec to < 150 ms.
>
>
>
> I know this probably not enough diagnostics to pin-point a cause. What
> explanations are these for this behavior?
>
>
>
> Misconfigured client connections?
>
>
>
> The seed node EC2 instance tested fine for general health.
>
>
>
> During the load test, this node had a ‘repair -pr’ job running on it which
> finished successfully, but read latency did not improve afterwards.
>
>
>
>
>
> ----------------
> Thank you
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>

Mime
View raw message