cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Clément Lardeur (JIRA) <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-11117) ColUpdateTimeDeltaHistogram histogram overflow
Date Mon, 22 Aug 2016 08:06:21 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15430271#comment-15430271
] 

Clément Lardeur edited comment on CASSANDRA-11117 at 8/22/16 8:06 AM:
----------------------------------------------------------------------

Hi we have the same problem with the version 3.0.8, the graphite reporter stop working like
CASSANDRA-12185.

{noformat}
ERROR [metrics-graphite-reporter-1-thread-1] 2016-08-22 09:40:42,618 ScheduledReporter.java:119
- RuntimeException thrown from GraphiteReporter#report. Exception was suppressed.
java.lang.IllegalStateException: Unable to compute ceiling for max when histogram overflowed
        at org.apache.cassandra.utils.EstimatedHistogram.rawMean(EstimatedHistogram.java:231)
~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.metrics.EstimatedHistogramReservoir$HistogramSnapshot.getMean(EstimatedHistogramReservoir.java:103)
~[apache-cassandra-3.0.8.jar:3.0.8]
        at com.codahale.metrics.graphite.GraphiteReporter.reportHistogram(GraphiteReporter.java:265)
~[metrics-graphite-3.1.2.jar:3.1.2]
        at com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:179)
~[metrics-graphite-3.1.2.jar:3.1.2]
        at com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162) ~[metrics-core-3.1.0.jar:3.1.0]
        at com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117) ~[metrics-core-3.1.0.jar:3.1.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_101]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_101]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_101]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
{noformat}

I don't think that ticket should be flagged as Minor because it's impacting our production
monitoring of the cluster. Since the version 3.x with the incompatibility of OpsCenter, many
users switch to another monitoring solution so the graphite reporter is now more critical
for the Open Source community than previously.

If I could help I can give you more details with our cluster settings.


was (Author: clardeur):
Hi we have the same problem with the version 3.0.8, the graphite reporter stop working like
CASSANDRA-12185.

{noformat}
ERROR [metrics-graphite-reporter-1-thread-1] 2016-08-22 09:40:42,618 ScheduledReporter.java:119
- RuntimeException thrown from GraphiteReporter#report. Exception was suppressed.
java.lang.IllegalStateException: Unable to compute ceiling for max when histogram overflowed
        at org.apache.cassandra.utils.EstimatedHistogram.rawMean(EstimatedHistogram.java:231)
~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.metrics.EstimatedHistogramReservoir$HistogramSnapshot.getMean(EstimatedHistogramReservoir.java:103)
~[apache-cassandra-3.0.8.jar:3.0.8]
        at com.codahale.metrics.graphite.GraphiteReporter.reportHistogram(GraphiteReporter.java:265)
~[metrics-graphite-3.1.2.jar:3.1.2]
        at com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:179)
~[metrics-graphite-3.1.2.jar:3.1.2]
        at com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162) ~[metrics-core-3.1.0.jar:3.1.0]
        at com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117) ~[metrics-core-3.1.0.jar:3.1.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_101]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_101]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_101]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
{noformat}

I don't think that ticket should be flagged as Minor because it's impacting our production
monitoring of the cluster. Since the version 3.x with the incompatibility of OpsCenter, many
users switch to another monitoring solution so the graphite reporter is now more critical
for the OOS community than previously.

If I could help I can give you more details with our cluster settings.

> ColUpdateTimeDeltaHistogram histogram overflow
> ----------------------------------------------
>
>                 Key: CASSANDRA-11117
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11117
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Chris Lohfink
>            Assignee: Joel Knighton
>            Priority: Minor
>             Fix For: 2.2.x, 3.0.x, 3.x
>
>
> {code}
> getting attribute Mean of org.apache.cassandra.metrics:type=ColumnFamily,name=ColUpdateTimeDeltaHistogram
threw an exceptionjavax.management.RuntimeMBeanException: java.lang.IllegalStateException:
Unable to compute ceiling for max when histogram overflowed
> {code}
> Although the fact that this histogram has 164 buckets already, I wonder if there is something
weird with the computation thats causing this to be so large? It appears to be coming from
updates to system.local
> {code}
> org.apache.cassandra.metrics:type=Table,keyspace=system,scope=local,name=ColUpdateTimeDeltaHistogram
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message