cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "n0rad (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-11595) Cassandra cannot start because of empty commitlog
Date Mon, 18 Apr 2016 12:35:25 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245584#comment-15245584
] 

n0rad commented on CASSANDRA-11595:
-----------------------------------

it's mixed up with gc but here is the final logs of the process. After that cassandra restart

{code}
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:163)
~[apache-cassandra-3.0.5.jar:3.0.5]
Metaspace       used 40691K, capacity 42565K, committed 43164K, reserved 1087488K
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
at org.apache.cassandra.db.commitlog.MemoryMappedSegment.<init>(MemoryMappedSegment.java:47)
~[apache-cassandra-3.0.5.jar:3.0.5]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [apache-cassandra-3.0.5.jar:3.0.5]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77-Debian]
class space    used 4325K, capacity 4685K, committed 4856K, reserved 1048576K
at org.apache.cassandra.db.commitlog.MemoryMappedSegment.createBuffer(MemoryMappedSegment.java:62)
at org.apache.cassandra.db.commitlog.CommitLogSegment.createSegment(CommitLogSegment.java:122)
~[apache-cassandra-3.0.5.jar:3.0.5]
at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:163)
at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$1.runMayThrow(CommitLogSegmentManager.java:122)
~[apache-cassandra-3.0.5.jar:3.0.5]
at org.apache.cassandra.db.commitlog.MemoryMappedSegment.<init>(MemoryMappedSegment.java:47)
ERROR 07:06:06 JVM state determined to be unstable.  Exiting forcefully due to:
Heap
at org.apache.cassandra.db.commitlog.CommitLogSegment.createSegment(CommitLogSegment.java:122)
java.io.FileNotFoundException: /data/commitlog/CommitLog-6-1460633590316.log (Too many open
files)
par new generation   total 1887488K, used 711037K [0x00000005c0000000, 0x0000000640000000,
0x0000000640000000)
at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$1.runMayThrow(CommitLogSegmentManager.java:122)
at java.io.RandomAccessFile.open0(Native Method) ~[na:1.8.0_77-Debian]
eden space 1677824K,  40% used [0x00000005c0000000, 0x00000005e9cd77d0, 0x0000000626680000)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316) ~[na:1.8.0_77-Debian]
from space 209664K,  12% used [0x0000000633340000, 0x0000000634cc7ff8, 0x0000000640000000)
java.io.FileNotFoundException: /data/commitlog/CommitLog-6-1460633590316.log (Too many open
files)
at java.lang.Thread.run(Thread.java:745)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) ~[na:1.8.0_77-Debian]
to   space 209664K,   0% used [0x0000000626680000, 0x0000000626680000, 0x0000000633340000)
at java.io.RandomAccessFile.open0(Native Method)
at org.apache.cassandra.db.commitlog.MemoryMappedSegment.createBuffer(MemoryMappedSegment.java:62)
~[apache-cassandra-3.0.5.jar:3.0.5]
concurrent mark-sweep generation total 6291456K, used 4072089K [0x0000000640000000, 0x00000007c0000000,
0x00000007c0000000)
{code}

> Cassandra cannot start because of empty commitlog
> -------------------------------------------------
>
>                 Key: CASSANDRA-11595
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11595
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: n0rad
>            Assignee: Benjamin Lerer
>
> After the crash of CASSANDRA-11594.
> Cassandra try to restart and fail because of commit log replay.
> Same on 4 of the crashed nodes out of 6.
> ```
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Could not
read commit log descriptor in file /data/commitlog/CommitLog-6-1460632496764.log
> at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:644)
[apache-cassandra-3.0.5.jar:3.0.5]
> ```
> This file is empty and is not the commitlog with the latest date.
> ```
> ...
> -rw-r--r-- 1 root root 32M Apr 16 21:46 CommitLog-6-1460632496761.log
> -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496762.log
> -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496763.log
> -rw-r--r-- 1 root root   0 Apr 16 21:47 CommitLog-6-1460632496764.log
> -rw-r--r-- 1 root root 32M Apr 16 21:50 CommitLog-6-1460843401097.log
> -rw-r--r-- 1 root root 32M Apr 16 21:51 CommitLog-6-1460843513346.log
> -rw-r--r-- 1 root root 32M Apr 16 21:53 CommitLog-6-1460843619271.log
> -rw-r--r-- 1 root root 32M Apr 16 21:55 CommitLog-6-1460843730533.log
> -rw-r--r-- 1 root root 32M Apr 16 21:57 CommitLog-6-1460843834129.log
> -rw-r--r-- 1 root root 32M Apr 16 21:58 CommitLog-6-1460843935094.log
> -rw-r--r-- 1 root root 32M Apr 16 22:00 CommitLog-6-1460844038543.log
> -rw-r--r-- 1 root root 32M Apr 16 22:02 CommitLog-6-1460844141003.log
> ...
> ```



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message