cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Per Otterström (JIRA) <>
Subject [jira] [Commented] (CASSANDRA-11752) histograms/metrics in 2.2 do not appear recency biased
Date Tue, 30 Aug 2016 07:30:21 GMT


Per Otterström commented on CASSANDRA-11752:

Interesting comparison. And I'm a bit surprised by the results. I would expect the EH to exaggerate
peaks rather than the other way around as it registers measurements in fixed value buckets
rounding _upwards_.

I wonder if this could be an effect of the fact that the EDR is based on samples. This means
that sometimes it can miss an outlier, but could this also mean that if an outlier is caught
it can be over-represented in the histogram?

Another explanation could be that the decay is to quick or slow in the DEHR (half-time set
to one minute right now). How frequently do you take readings?

> histograms/metrics in 2.2 do not appear recency biased
> ------------------------------------------------------
>                 Key: CASSANDRA-11752
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Chris Burroughs
>            Assignee: Per Otterström
>              Labels: metrics
>             Fix For: 2.2.8, 3.0.9, 3.8
>         Attachments: 11752-2.2-v2.txt, 11752-2.2-v2b.txt, 11752-2.2.txt, boost-metrics.png,
c-jconsole-comparison.png, c-metrics.png, default-histogram.png, server-patch-v2.png
> In addition to upgrading to metrics3, CASSANDRA-5657 switched to using  a custom histogram
implementation.  After upgrading to Cassandra 2.2 histograms/timer metrics are not suspiciously
flat.  To be useful for graphing and alerting metrics need to be biased towards recent events.
> I have attached images that I think illustrate this.
>  * The first two are a comparison between latency observed by a C* 2.2 (us) cluster shoring
very flat lines and a client (using metrics 2.2.0, ms) showing server performance problems.
 We can't rule out with total certainty that something else isn't the cause (that's why we
measure from both the client & server) but they very rarely disagree.
>  * The 3rd image compares jconsole viewing of metrics on a 2.2 and 2.1 cluster over several
minutes.  Not a single digit changed on the 2.2 cluster.

This message was sent by Atlassian JIRA

View raw message