cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefano Ortolani <ostef...@gmail.com>
Subject Upgrade from 2.1.11 to 3.0.5 leads to unstable nodes
Date Fri, 06 May 2016 00:01:45 GMT
Hi,

I am experiencing some weird behaviors after upgrading 2 nodes (out of 13)
to C* 3.0.5 (from 2.1.11). Basically, after restarting a second time, there
is a small chance that the node will die without outputting anything to the
logs (not even dmesg).

This happened on both nodes I upgraded. The only "anomalies" I see in the
logs (although not related to the moment a node dies) are:

* Lots of the following messages against all IPs of the cluster (every
second)

DEBUG [GossipStage:1] 2016-05-05 23:52:02,260 FailureDetector.java:456 -
Ignoring interval time of 2540341017 for /x.y.b.5
DEBUG [GossipStage:1] 2016-05-05 23:52:02,260 FailureDetector.java:456 -
Ignoring interval time of 2000551507 for /x.y.a.7
DEBUG [GossipStage:1] 2016-05-05 23:52:02,260 FailureDetector.java:456 -
Ignoring interval time of 2000479104 for /x.y.a.3
DEBUG [GossipStage:1] 2016-05-05 23:52:02,260 FailureDetector.java:456 -
Ignoring interval time of 2000471247 for /x.y.b.3
DEBUG [GossipStage:1] 2016-05-05 23:52:03,259 FailureDetector.java:456 -
Ignoring interval time of 2000605748 for /x.y.a.5
DEBUG [GossipStage:1] 2016-05-05 23:52:03,260 FailureDetector.java:456 -
Ignoring interval time of 2000731307 for /x.y.b.6
DEBUG [GossipStage:1] 2016-05-05 23:52:03,260 FailureDetector.java:456 -
Ignoring interval time of 3000404107 for /x.y.b.1

* Some metrics are not being pushed to graphite (but some do get to the
server). Also, every time the node tries to push them I can see the
following error in the logs:

ERROR [metrics-graphite-reporter-1-thread-1] 2016-05-05 23:53:37,770
ScheduledReporter.java:119 - RuntimeException thrown from
GraphiteReporter#report. Exception was suppressed.
java.lang.IllegalStateException: Unable to compute ceiling for max when
histogram overflowed
at
org.apache.cassandra.utils.EstimatedHistogram.rawMean(EstimatedHistogram.java:231)
~[apache-cassandra-3.0.5.jar:3.0.5]
at
org.apache.cassandra.metrics.EstimatedHistogramReservoir$HistogramSnapshot.getMean(EstimatedHistogramReservoir.java:103)
~[apache-cassandra-3.0.5.jar:3.0.5]
at
com.codahale.metrics.graphite.GraphiteReporter.reportHistogram(GraphiteReporter.java:252)
~[metrics-graphite-3.1.0.jar:3.1.0]
at
com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:166)
~[metrics-graphite-3.1.0.jar:3.1.0]
at
com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162)
~[metrics-core-3.1.0.jar:3.1.0]
at com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117)
~[metrics-core-3.1.0.jar:3.1.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_60]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]

Besides these, logs are clean. I've opened a ticket here (
https://issues.apache.org/jira/browse/CASSANDRA-11723) but any help
debugging this is more than welcome.

Regards,
Stefano

Mime
View raw message