cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeff Jirsa (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-12844) nodetool drain causing mutiple nodes crashing with hint file corruption in Cassandra 3.9
Date Wed, 26 Oct 2016 19:35:58 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-12844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jeff Jirsa updated CASSANDRA-12844:
-----------------------------------
    Component/s:     (was: Tools)
                 Core

> nodetool drain causing mutiple nodes crashing with hint file corruption in Cassandra
3.9
> ----------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-12844
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12844
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Harikrishnan
>
> The steps are as follows.
> we have 4/4 node cassandra running in 3.9 version.
> In one node made some changes to cassanra.yaml. issued a nodetool drain 
> killed the cassandra process and restarted the node. After sometime nodetool status reported
multiple nodes are down in that DC.
> Went and check the system.log of all the files and found the hint corruption occuring(CASSANDRA-12728).
 nodetool drain causing this corruption and bringing multiple nodes down is a big concern.
> ERROR [HintsDispatcher:2] 2016-10-26 12:17:59,361 HintsDispatchExecutor.java:225 - Failed
to dispatch hints file 4d1362f0-053c-4042-80a7-bfc85a26c90f-1477509190999-1.hints: file is
corrupted ({})
> org.apache.cassandra.io.FSReadError: java.io.EOFException
>         at org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:284)
~[apache-cassandra-3.9.jar:3.9]
>         at org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:254)
~[apache-cassandra-3.9.jar:3.9]
>         at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.9.jar:3.9]
>         at org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156)
~[apache-cassandra-3.9.jar:3.9]
>         at org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137)
~[apache-cassandra-3.9.jar:3.9]
>         at org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119)
~[apache-cassandra-3.9.jar:3.9]
>         at org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91)
~[apache-cassandra-3.9.jar:3.9]
>         at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:259)
[apache-cassandra-3.9.jar:3.9]
>         at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242)
[apache-cassandra-3.9.jar:3.9]
>         at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:220)
[apache-cassandra-3.9.jar:3.9]
>         at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:199)
[apache-cassandra-3.9.jar:3.9]
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_102]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_102]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_102]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_102]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message