cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Youcef HILEM (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-10292) java.lang.AssertionError: attempted to delete non-existing file CommitLog...
Date Sat, 07 May 2016 07:23:12 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275137#comment-15275137
] 

Youcef HILEM edited comment on CASSANDRA-10292 at 5/7/16 7:22 AM:
------------------------------------------------------------------

We have encountered the same problem. The node has stopped.
Environnement :
- CentOS Linux release 7.2.1511
- java version "1.7.0_79"   (build 1.7.0_79-b15)
- apache-cassandra-2.1.9

Stack trace :

ERROR [COMMIT-LOG-ALLOCATOR] 2016-05-07 03:32:59,499 StorageService.java:453 - org.apache.cassandra.service.StorageService
- Stopping gossiper
WARN  [COMMIT-LOG-ALLOCATOR] 2016-05-07 03:32:59,500 StorageService.java:359 - org.apache.cassandra.service.StorageService
- Stopping gossip by operator request
INFO  [COMMIT-LOG-ALLOCATOR] 2016-05-07 03:32:59,500 Gossiper.java:1437 - org.apache.cassandra.gms.Gossiper
- Announcing shutdown
INFO  [MigrationStage:1] 2016-05-07 03:32:59,969 QueryProcessor.java:632 - org.apache.cassandra.cql3.QueryProcessor
- Table smoketest_lpa2e025_svc.smoketest was dropped, invalidating related prepared statements
ERROR [COMMIT-LOG-ALLOCATOR] 2016-05-07 03:33:01,502 StorageService.java:458 - org.apache.cassandra.service.StorageService
- Stopping RPC server
INFO  [COMMIT-LOG-ALLOCATOR] 2016-05-07 03:33:01,503 ThriftServer.java:142 - org.apache.cassandra.thrift.ThriftServer
- Stop listening to thrift clients
ERROR [COMMIT-LOG-ALLOCATOR] 2016-05-07 03:33:01,513 StorageService.java:463 - org.apache.cassandra.service.StorageService
- Stopping native transport
INFO  [COMMIT-LOG-ALLOCATOR] 2016-05-07 03:33:01,554 Server.java:213 - org.apache.cassandra.transport.Server
- Stop listening for CQL clients
ERROR [COMMIT-LOG-ALLOCATOR] 2016-05-07 03:33:01,555 CommitLog.java:397 - org.apache.cassandra.db.commitlog.CommitLog
- Failed managing commit log segments. Commit disk failure policy is stop; terminating thread
java.lang.AssertionError: attempted to delete non-existing file CommitLog-4-1462375290205.log
	at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:126) ~[apache-cassandra-2.1.9.jar:2.1.9]
	at org.apache.cassandra.db.commitlog.CommitLogSegment.delete(CommitLogSegment.java:343) ~[apache-cassandra-2.1.9.jar:2.1.9]
	at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$5.call(CommitLogSegmentManager.java:418)
~[apache-cassandra-2.1.9.jar:2.1.9]
	at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$5.call(CommitLogSegmentManager.java:413)
~[apache-cassandra-2.1.9.jar:2.1.9]
	at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$1.runMayThrow(CommitLogSegmentManager.java:152)
~[apache-cassandra-2.1.9.jar:2.1.9]
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [apache-cassandra-2.1.9.jar:2.1.9]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]



was (Author: yhilem):
We have encountered the same problem. The node has stopped.
Environnement :
- CentOS Linux release 7.2.1511
- java version "1.7.0_79"   (build 1.7.0_79-b15)

Stack trace :

ERROR [COMMIT-LOG-ALLOCATOR] 2016-05-07 03:32:59,499 StorageService.java:453 - org.apache.cassandra.service.StorageService
- Stopping gossiper
WARN  [COMMIT-LOG-ALLOCATOR] 2016-05-07 03:32:59,500 StorageService.java:359 - org.apache.cassandra.service.StorageService
- Stopping gossip by operator request
INFO  [COMMIT-LOG-ALLOCATOR] 2016-05-07 03:32:59,500 Gossiper.java:1437 - org.apache.cassandra.gms.Gossiper
- Announcing shutdown
INFO  [MigrationStage:1] 2016-05-07 03:32:59,969 QueryProcessor.java:632 - org.apache.cassandra.cql3.QueryProcessor
- Table smoketest_lpa2e025_svc.smoketest was dropped, invalidating related prepared statements
ERROR [COMMIT-LOG-ALLOCATOR] 2016-05-07 03:33:01,502 StorageService.java:458 - org.apache.cassandra.service.StorageService
- Stopping RPC server
INFO  [COMMIT-LOG-ALLOCATOR] 2016-05-07 03:33:01,503 ThriftServer.java:142 - org.apache.cassandra.thrift.ThriftServer
- Stop listening to thrift clients
ERROR [COMMIT-LOG-ALLOCATOR] 2016-05-07 03:33:01,513 StorageService.java:463 - org.apache.cassandra.service.StorageService
- Stopping native transport
INFO  [COMMIT-LOG-ALLOCATOR] 2016-05-07 03:33:01,554 Server.java:213 - org.apache.cassandra.transport.Server
- Stop listening for CQL clients
ERROR [COMMIT-LOG-ALLOCATOR] 2016-05-07 03:33:01,555 CommitLog.java:397 - org.apache.cassandra.db.commitlog.CommitLog
- Failed managing commit log segments. Commit disk failure policy is stop; terminating thread
java.lang.AssertionError: attempted to delete non-existing file CommitLog-4-1462375290205.log
	at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:126) ~[apache-cassandra-2.1.9.jar:2.1.9]
	at org.apache.cassandra.db.commitlog.CommitLogSegment.delete(CommitLogSegment.java:343) ~[apache-cassandra-2.1.9.jar:2.1.9]
	at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$5.call(CommitLogSegmentManager.java:418)
~[apache-cassandra-2.1.9.jar:2.1.9]
	at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$5.call(CommitLogSegmentManager.java:413)
~[apache-cassandra-2.1.9.jar:2.1.9]
	at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$1.runMayThrow(CommitLogSegmentManager.java:152)
~[apache-cassandra-2.1.9.jar:2.1.9]
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [apache-cassandra-2.1.9.jar:2.1.9]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]


> java.lang.AssertionError: attempted to delete non-existing file CommitLog...
> ----------------------------------------------------------------------------
>
>                 Key: CASSANDRA-10292
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10292
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: CentOS Linux 7.1.1503, Cassandra 2.1.8 stable version, 6 nodes cluster
>            Reporter: Dawid Szejnfeld
>            Priority: Critical
>
> From time to time some nodes are stopping to work due to error in logs like this:
> INFO  [CompactionExecutor:2475] 2015-09-09 12:36:50,363 CompactionTask.java:274 - Compacted
4 sstables to [/mnt/cassandra--storage-machine/data/system/compactions_in_progress-55080ab05d9c38
> 8690a4acb25fe1f77b/system-compactions_in_progress-ka-126,].  419 bytes to 42 (~10% of
original) in 33ms = 0.001214MB/s.  4 total partitions merged to 1.  Partition merge counts
were {2:2, }
> INFO  [COMMIT-LOG-ALLOCATOR] 2015-09-09 12:52:34,166 ColumnFamilyStore.java:912 - Enqueuing
flush of settings: 78364 (0%) on-heap, 0 (0%) off-heap
> INFO  [MemtableFlushWriter:301] 2015-09-09 12:52:34,172 Memtable.java:347 - Writing Memtable-settings@1126939979(0.113KiB
serialized bytes, 1850 ops, 0%/0% of on/off-heap limit)
> INFO  [MemtableFlushWriter:301] 2015-09-09 12:52:34,174 Memtable.java:382 - Completed
flushing /mnt/cassandra--storage-machine/data/OpsCenter/settings-464866c04b1311e590698d1a9fd4ba8b/OpsCe
> nter-settings-tmp-ka-12-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1441362636571,
position=33554415)
> ERROR [COMMIT-LOG-ALLOCATOR] 2015-09-09 12:52:34,194 StorageService.java:453 - Stopping
gossiper
> WARN  [COMMIT-LOG-ALLOCATOR] 2015-09-09 12:52:34,195 StorageService.java:359 - Stopping
gossip by operator request
> INFO  [COMMIT-LOG-ALLOCATOR] 2015-09-09 12:52:34,195 Gossiper.java:1410 - Announcing
shutdown
> ERROR [COMMIT-LOG-ALLOCATOR] 2015-09-09 12:52:36,195 StorageService.java:458 - Stopping
RPC server
> INFO  [COMMIT-LOG-ALLOCATOR] 2015-09-09 12:52:36,196 ThriftServer.java:142 - Stop listening
to thrift clients
> ERROR [COMMIT-LOG-ALLOCATOR] 2015-09-09 12:52:36,204 StorageService.java:463 - Stopping
native transport
> INFO  [COMMIT-LOG-ALLOCATOR] 2015-09-09 12:52:36,422 Server.java:213 - Stop listening
for CQL clients
> ERROR [COMMIT-LOG-ALLOCATOR] 2015-09-09 12:52:36,423 CommitLog.java:397 - Failed managing
commit log segments. Commit disk failure policy is stop; terminating thread
> java.lang.AssertionError: attempted to delete non-existing file CommitLog-4-1441362636316.log
>         at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:126)
~[apache-cassandra-2.1.8.jar:2.1.8]
>         at org.apache.cassandra.db.commitlog.CommitLogSegment.delete(CommitLogSegment.java:343)
~[apache-cassandra-2.1.8.jar:2.1.8]
>         at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$5.call(CommitLogSegmentManager.java:418)
~[apache-cassandra-2.1.8.jar:2.1.8]
>         at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$5.call(CommitLogSegmentManager.java:413)
~[apache-cassandra-2.1.8.jar:2.1.8]
>         at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$1.runMayThrow(CommitLogSegmentManager.java:152)
~[apache-cassandra-2.1.8.jar:2.1.8]
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [apache-cassandra-2.1.8.jar:2.1.8]
>         at java.lang.Thread.run(Thread.java:745) [na:1.7.0_85]
> After I create missing commit log file and restart cassandra service everything is OK
then.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message