cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joshua McKenzie (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-8743) Repair on NFS in version 2.1.2
Date Mon, 16 Mar 2015 21:17:38 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-8743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14363965#comment-14363965
] 

Joshua McKenzie commented on CASSANDRA-8743:
--------------------------------------------

[~tamarn] No luck reproducing on this side on 2.1.2.

3-node cluster with all files stored over NFS mounted drive, killed a node in the middle of
a long-running stress and cleared its data, brought it back up and repaired on it. Doing this
repeatedly and not seeing any exceptions.

If you can consistently reproduce this, if you could try out 2.1-HEAD or even better, trunk,
that would be very helpful. Also, if you could provide more information about your use case
and topology that would help as well.

> Repair on NFS in version 2.1.2
> ------------------------------
>
>                 Key: CASSANDRA-8743
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8743
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Tamar Nirenberg
>            Assignee: Joshua McKenzie
>            Priority: Minor
>
> Running repair over NFS in Cassandra 2.1.2 encounters this error and crashes the ring:
> ERROR [ValidationExecutor:2] 2015-01-22 11:48:14,811 Validator.java:232 - Failed creating
a merkle tree for [repair #c84c7c70-a21b-11e4-aeca-19e6d7fa2595 on ATTRIBUTES/LINKS, (11621838520493020277529637175352775759,11853478749048239324667887059881170862]],
/10.1.234.63 (see log for details)
> ERROR [ValidationExecutor:2] 2015-01-22 11:48:14,827 CassandraDaemon.java:153 - Exception
in thread Thread[ValidationExecutor:2,1,main]
> org.apache.cassandra.io.FSWriteError: java.nio.file.DirectoryNotEmptyException: /exlibris/cassandra/local/data/data/ATTRIBUTES/LINKS/snapshots/c84c7c70-a21b-11e4-aeca-19e6d7fa2595
>         at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:135)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:381)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.Directories.clearSnapshot(Directories.java:547) ~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.ColumnFamilyStore.clearSnapshot(ColumnFamilyStore.java:2223)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:939)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:97)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:557)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_71]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_71]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_71]
>         at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> Caused by: java.nio.file.DirectoryNotEmptyException: /exlibris/cassandra/local/data/data/ATTRIBUTES/LINKS/snapshots/c84c7c70-a21b-11e4-aeca-19e6d7fa2595
>         at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
~[na:1.7.0_71]
>         at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
~[na:1.7.0_71]
>         at java.nio.file.Files.delete(Files.java:1079) ~[na:1.7.0_71]
>         at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:131)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         ... 10 common frames omitted
> ERROR [ValidationExecutor:2] 2015-01-22 11:48:14,829 StorageService.java:383 - Stopping
gossiper
> WARN  [ValidationExecutor:2] 2015-01-22 11:48:14,829 StorageService.java:291 - Stopping
gossip by operator request
> INFO  [ValidationExecutor:2] 2015-01-22 11:48:14,829 Gossiper.java:1318 - Announcing
shutdown



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message