cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "T Jake Luciani (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-9973) java.lang.IllegalStateException: Unable to compute when histogram overflowed
Date Mon, 10 Aug 2015 14:58:46 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-9973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14680226#comment-14680226
] 

T Jake Luciani commented on CASSANDRA-9973:
-------------------------------------------

This looks like a regression from CASSANDRA-5657 where we replaced the default metrics histogram
with our estimated histogram.  The histogram is padded to support values upto 12B but looks
like readlatency is kept in nanoseconds which is cutting it too close.  So we should pad this
to handle 30 seconds of nanoseconds? Patch attached.



> java.lang.IllegalStateException: Unable to compute when histogram overflowed
> ----------------------------------------------------------------------------
>
>                 Key: CASSANDRA-9973
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9973
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Mark Manley
>            Assignee: Carl Yeksigian
>             Fix For: 2.2.x
>
>
> I recently, and probably mistakenly, upgraded one of my production C* clusters to 2.2.0.
 I am seeing these errors in the logs, followed by an intense period of garbage collection
until the node, then the ring, becomes crippled:
> {code}
> ERROR [OptionalTasks:1] 2015-08-04 03:24:56,057 CassandraDaemon.java:182 - Exception
in thread Thread[OptionalTasks:1,5,main]
> java.lang.IllegalStateException: Unable to compute when histogram overflowed
>         at org.apache.cassandra.utils.EstimatedHistogram.percentile(EstimatedHistogram.java:179)
~[apache-cassandra-2.2.0.jar:2.2.0]
>         at org.apache.cassandra.metrics.EstimatedHistogramReservoir$HistogramSnapshot.getValue(EstimatedHistogramReservoir.java:84)
~[apache-cassandra-2.2.0.jar:2.2.0]
>         at org.apache.cassandra.db.ColumnFamilyStore$3.run(ColumnFamilyStore.java:405)
~[apache-cassandra-2.2.0.jar:2.2.0]
>         at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118)
~[apache-cassandra-2.2.0.jar:2.2.0]
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45]
>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_45]
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_45]
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_45]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_45]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> {code}
> I am not sure if the GC instability is this or something else, but I though this histogram
overflow issue was fixed in 2.1.3?  Anyway, reporting now as a possible regression.  Please
let me know what I can provide in terms of information to help with this.  Thanks!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message