cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Lohfink (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-11752) histograms/metrics in 2.2 do not appear recency biased
Date Wed, 11 May 2016 21:26:13 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15280857#comment-15280857
] 

Chris Lohfink commented on CASSANDRA-11752:
-------------------------------------------

ExponentiallyDecayingReservoir is based on idea of that the data has a normal distribution,
in latencies with long tails (ie GCs) the max/99th quickly becomes very inaccurate as more
and more data goes through it. 

The EH does not by itself provide a "recent" view but you can derive it easily enough. Probably
a change to hdr histogram, or adding an exp decay to the EH, or exposing the clear operation
on the mbean (reset when you read them) would help. but using random sampling for latencies
is the wrong direction imho.

> histograms/metrics in 2.2 do not appear recency biased
> ------------------------------------------------------
>
>                 Key: CASSANDRA-11752
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11752
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Chris Burroughs
>              Labels: metrics
>         Attachments: boost-metrics.png, c-jconsole-comparison.png, c-metrics.png, default-histogram.png
>
>
> In addition to upgrading to metrics3, CASSANDRA-5657 switched to using  a custom histogram
implementation.  After upgrading to Cassandra 2.2 histograms/timer metrics are not suspiciously
flat.  To be useful for graphing and alerting metrics need to be biased towards recent events.
> I have attached images that I think illustrate this.
>  * The first two are a comparison between latency observed by a C* 2.2 (us) cluster shoring
very flat lines and a client (using metrics 2.2.0, ms) showing server performance problems.
 We can't rule out with total certainty that something else isn't the cause (that's why we
measure from both the client & server) but they very rarely disagree.
>  * The 3rd image compares jconsole viewing of metrics on a 2.2 and 2.1 cluster over several
minutes.  Not a single digit changed on the 2.2 cluster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message