cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amit Khare (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-9525) Commitlog allocation failure doesn't stop the entire node.
Date Mon, 13 Jul 2015 12:55:05 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-9525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14624607#comment-14624607
] 

Amit Khare commented on CASSANDRA-9525:
---------------------------------------

We are facing the same issue in 2.1.3 version as well. The node remains in the hang state
and does not serve any writes going forward. Also, causes memtable writes to be blocked.



> Commitlog allocation failure doesn't stop the entire node.
> ----------------------------------------------------------
>
>                 Key: CASSANDRA-9525
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9525
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Jeremiah Jordan
>             Fix For: 2.1.x
>
>
> If there is an error opening a commit log segment, the thread dies, but it doesn't stop
the whole node.
> Got the following on a node:
> {noformat}
> ERROR [COMMIT-LOG-ALLOCATOR] 2015-05-29 17:06:45,722  CommitLog.java:397 - Failed managing
commit log segments. Commit disk failure policy is stop; terminating thread
> org.apache.cassandra.io.FSWriteError: java.io.FileNotFoundException: /var/lib/cassandra/commitlog/CommitLog-4-1432937194590.log
(Permission denied)
> 	at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:177)
~[cassandra-all-2.1.5.jar:2.1.5]
> 	at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$4.call(CommitLogSegmentManager.java:397)
~[cassandra-all-2.1.5.jar:2.1.5]
> 	at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$4.call(CommitLogSegmentManager.java:394)
~[cassandra-all-2.1.5.jar:2.1.5]
> 	at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$1.runMayThrow(CommitLogSegmentManager.java:152)
~[cassandra-all-2.1.5.jar:2.1.5]
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [cassandra-all-2.1.5.jar:2.1.5]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> Caused by: java.io.FileNotFoundException: /var/lib/cassandra/commitlog/CommitLog-4-1432937194590.log
(Permission denied)
> 	at java.io.RandomAccessFile.open0(Native Method) ~[na:1.8.0_45]
> 	at java.io.RandomAccessFile.open(RandomAccessFile.java:316) ~[na:1.8.0_45]
> 	at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) ~[na:1.8.0_45]
> 	at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:155)
~[cassandra-all-2.1.5.jar:2.1.5]
> 	... 5 common frames omitted
> {noformat}
> And the node stayed kind of up, didn't notice something wrong until the node died OOM
because some threads were dead and others weren't.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message