cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harikrishnan (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-12844) nodetool drain causing mutiple nodes crashing with hint file corruption in Cassandra 3.9
Date Wed, 26 Oct 2016 19:27:58 GMT
Harikrishnan created CASSANDRA-12844:
----------------------------------------

             Summary: nodetool drain causing mutiple nodes crashing with hint file corruption
in Cassandra 3.9
                 Key: CASSANDRA-12844
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12844
             Project: Cassandra
          Issue Type: Bug
          Components: Tools
            Reporter: Harikrishnan


The steps are as follows.
we have 4/4 node cassandra running in 3.9 version.
In one node made some changes to cassanra.yaml. issued a nodetool drain 
killed the cassandra process and restarted the node. After sometime nodetool status reported
multiple nodes are down in that DC.

Went and check the system.log of all the files and found the hint corruption occuring(CASSANDRA-12728).
 nodetool drain causing this corruption and bringing multiple nodes down is a big concern.




ERROR [HintsDispatcher:2] 2016-10-26 12:17:59,361 HintsDispatchExecutor.java:225 - Failed
to dispatch hints file 4d1362f0-053c-4042-80a7-bfc85a26c90f-1477509190999-1.hints: file is
corrupted ({})
org.apache.cassandra.io.FSReadError: java.io.EOFException
        at org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:284)
~[apache-cassandra-3.9.jar:3.9]
        at org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:254)
~[apache-cassandra-3.9.jar:3.9]
        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.9.jar:3.9]
        at org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156)
~[apache-cassandra-3.9.jar:3.9]
        at org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137)
~[apache-cassandra-3.9.jar:3.9]
        at org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119) ~[apache-cassandra-3.9.jar:3.9]
        at org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91) ~[apache-cassandra-3.9.jar:3.9]
        at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:259)
[apache-cassandra-3.9.jar:3.9]
        at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242)
[apache-cassandra-3.9.jar:3.9]
        at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:220)
[apache-cassandra-3.9.jar:3.9]
        at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:199)
[apache-cassandra-3.9.jar:3.9]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_102]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_102]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_102]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_102]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message