cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andreas Schnitzerling (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-6283) Windows 7 data files keept open / can't be deleted after compaction.
Date Fri, 29 Nov 2013 11:52:36 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13835321#comment-13835321
] 

Andreas Schnitzerling edited comment on CASSANDRA-6283 at 11/29/13 11:51 AM:
-----------------------------------------------------------------------------

I think I can cancel repair without -par. After repairing one keyspace I got al lot of errors
but not the leak-detect-messages. I think, not RAR is responsible during repair. The error
on the Validation Node:
{panel:title=system.log}
ERROR [ValidationExecutor:1] 2013-11-29 12:15:38,370 Validator.java (line 242) Failed creating
a merkle tree for [repair #92b6ccb0-58e7-11e3-aac3-b13a5fe180aa on nieste/niesteplants, (-5215786285174483271,-5206407297765302700]],
/10.6.8.78 (see log for details)
ERROR [ValidationExecutor:1] 2013-11-29 12:15:38,370 CassandraDaemon.java (line 187) Exception
in thread Thread[ValidationExecutor:1,1,main]
FSWriteError in D:\Programme\cassandra\data\nieste\niesteplants\snapshots\92b6ccb0-58e7-11e3-aac3-b13a5fe180aa\nieste-niesteplants-jb-19-Data.db
	at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:120)
	at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:382)
	at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:378)
	at org.apache.cassandra.db.Directories.clearSnapshot(Directories.java:416)
	at org.apache.cassandra.db.ColumnFamilyStore.clearSnapshot(ColumnFamilyStore.java:1801)
	at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:810)
	at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:62)
	at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:397)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: java.nio.file.FileSystemException: D:\Programme\cassandra\data\nieste\niesteplants\snapshots\92b6ccb0-58e7-11e3-aac3-b13a5fe180aa\nieste-niesteplants-jb-19-Data.db:
Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet
wird.

	at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
	at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
	at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
	at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
	at sun.nio.fs.AbstractFileSystemProvider.delete(Unknown Source)
	at java.nio.file.Files.delete(Unknown Source)
	at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:116)
	... 11 more
{panel}
The error on the repair-node:
{panel:title=system.log}
ERROR [AntiEntropySessions:1] 2013-11-29 12:15:38,419 RepairSession.java (line 278) [repair
#92b6ccb0-58e7-11e3-aac3-b13a5fe180aa] session completed with the following error
org.apache.cassandra.exceptions.RepairException: [repair #92b6ccb0-58e7-11e3-aac3-b13a5fe180aa
on nieste/niesteplants, (-5215786285174483271,-5206407297765302700]] Validation failed in
/10.9.9.69
	at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:152)
	at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:188)
	at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:59)
	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:60)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
ERROR [AntiEntropySessions:1] 2013-11-29 12:15:38,429 CassandraDaemon.java (line 187) Exception
in thread Thread[AntiEntropySessions:1,5,RMI Runtime]
java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException: [repair #92b6ccb0-58e7-11e3-aac3-b13a5fe180aa
on nieste/niesteplants, (-5215786285174483271,-5206407297765302700]] Validation failed in
/10.9.9.69
	at com.google.common.base.Throwables.propagate(Throwables.java:160)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.cassandra.exceptions.RepairException: [repair #92b6ccb0-58e7-11e3-aac3-b13a5fe180aa
on nieste/niesteplants, (-5215786285174483271,-5206407297765302700]] Validation failed in
/10.9.9.69
	at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:152)
	at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:188)
	at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:59)
	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:60)
	... 3 more
{panel}
{panel:title=NodeTool}
Starting NodeTool
[2013-11-29 12:15:37,329] Starting repair command #1, repairing 256 ranges for keyspace nieste
[2013-11-29 12:19:46,219] Repair session 92b6ccb0-58e7-11e3-aac3-b13a5fe180aa for range (-5215786285174483271,-5206407297765302700]
failed with error org.apache.cassandra.exceptions.RepairException: [repair #92b6ccb0-58e7-11e3-aac3-b13a5fe180aa
on nieste/niesteplants, (-5215786285174483271,-5206407297765302700]] Validation failed in
/10.9.9.69
[2013-11-29 12:19:46,219] Repair session 935b9830-58e7-11e3-aac3-b13a5fe180aa for range (8782854129978293476,8784492934430401685]
failed with error org.apache.cassandra.exceptions.RepairException: [repair #935b9830-58e7-11e3-aac3-b13a5fe180aa
on nieste/niesteplants, (8782854129978293476,8784492934430401685]] Validation failed in /10.9.9.69
[2013-11-29 12:19:46,219] Repair session 93b725b0-58e7-11e3-aac3-b13a5fe180aa for range (-5799639987122737930,-5786898749415113092]
failed with error org.apache.cassandra.exceptions.RepairException: [repair #93b725b0-58e7-11e3-aac3-b13a5fe180aa
on nieste/nfiles, (-5799639987122737930,-5786898749415113092]] Validation failed in /10.9.9.240
[2013-11-29 12:19:46,229] Repair session 94b16430-58e7-11e3-aac3-b13a5fe180aa for range (2664807773952357126,2669403005419855407]
failed with error org.apache.cassandra.exceptions.RepairException: [repair #94b16430-58e7-11e3-aac3-b13a5fe180aa
on nieste/nfiles, (2664807773952357126,2669403005419855407]] Validation failed in /10.9.9.240
[2013-11-29 12:19:46,229] Repair session 9503c9f0-58e7-11e3-aac3-b13a5fe180aa for range (-6420115574437655437,-6410524043851626540]
finished
{panel}
Im curious, if cache-file-deleting will write leak-messages...
I'm using C* 2.0.3-release with leak-detection.patch.


was (Author: andie78):
I think I can cancel repair without -par. After repairing one keyspace I got al lot of errors
but not the leak-detect-messages. I think, not RAR is responsible during repair. The error
on the Validation Node:
{panel:title=system.log}
ERROR [ValidationExecutor:1] 2013-11-29 12:15:38,370 Validator.java (line 242) Failed creating
a merkle tree for [repair #92b6ccb0-58e7-11e3-aac3-b13a5fe180aa on nieste/niesteplants, (-5215786285174483271,-5206407297765302700]],
/10.6.8.78 (see log for details)
ERROR [ValidationExecutor:1] 2013-11-29 12:15:38,370 CassandraDaemon.java (line 187) Exception
in thread Thread[ValidationExecutor:1,1,main]
FSWriteError in D:\Programme\cassandra\data\nieste\niesteplants\snapshots\92b6ccb0-58e7-11e3-aac3-b13a5fe180aa\nieste-niesteplants-jb-19-Data.db
	at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:120)
	at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:382)
	at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:378)
	at org.apache.cassandra.db.Directories.clearSnapshot(Directories.java:416)
	at org.apache.cassandra.db.ColumnFamilyStore.clearSnapshot(ColumnFamilyStore.java:1801)
	at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:810)
	at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:62)
	at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:397)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: java.nio.file.FileSystemException: D:\Programme\cassandra\data\nieste\niesteplants\snapshots\92b6ccb0-58e7-11e3-aac3-b13a5fe180aa\nieste-niesteplants-jb-19-Data.db:
Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet
wird.

	at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
	at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
	at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
	at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
	at sun.nio.fs.AbstractFileSystemProvider.delete(Unknown Source)
	at java.nio.file.Files.delete(Unknown Source)
	at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:116)
	... 11 more
{panel}
The error on the repair-node:
{panel:title=system.log}
Starting NodeTool
[2013-11-29 12:15:37,329] Starting repair command #1, repairing 256 ranges for keyspace nieste
[2013-11-29 12:19:46,219] Repair session 92b6ccb0-58e7-11e3-aac3-b13a5fe180aa for range (-5215786285174483271,-5206407297765302700]
failed with error org.apache.cassandra.exceptions.RepairException: [repair #92b6ccb0-58e7-11e3-aac3-b13a5fe180aa
on nieste/niesteplants, (-5215786285174483271,-5206407297765302700]] Validation failed in
/10.9.9.69
[2013-11-29 12:19:46,219] Repair session 935b9830-58e7-11e3-aac3-b13a5fe180aa for range (8782854129978293476,8784492934430401685]
failed with error org.apache.cassandra.exceptions.RepairException: [repair #935b9830-58e7-11e3-aac3-b13a5fe180aa
on nieste/niesteplants, (8782854129978293476,8784492934430401685]] Validation failed in /10.9.9.69
[2013-11-29 12:19:46,219] Repair session 93b725b0-58e7-11e3-aac3-b13a5fe180aa for range (-5799639987122737930,-5786898749415113092]
failed with error org.apache.cassandra.exceptions.RepairException: [repair #93b725b0-58e7-11e3-aac3-b13a5fe180aa
on nieste/nfiles, (-5799639987122737930,-5786898749415113092]] Validation failed in /10.9.9.240
[2013-11-29 12:19:46,229] Repair session 94b16430-58e7-11e3-aac3-b13a5fe180aa for range (2664807773952357126,2669403005419855407]
failed with error org.apache.cassandra.exceptions.RepairException: [repair #94b16430-58e7-11e3-aac3-b13a5fe180aa
on nieste/nfiles, (2664807773952357126,2669403005419855407]] Validation failed in /10.9.9.240
[2013-11-29 12:19:46,229] Repair session 9503c9f0-58e7-11e3-aac3-b13a5fe180aa for range (-6420115574437655437,-6410524043851626540]
finished
{panel}
Im curious, if cache-file-deleting will write leak-messages...
I'm using C* 2.0.3-release with leak-detection.patch.

> Windows 7 data files keept open / can't be deleted after compaction.
> --------------------------------------------------------------------
>
>                 Key: CASSANDRA-6283
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6283
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Windows 7 (32) / Java 1.7.0.45
>            Reporter: Andreas Schnitzerling
>            Priority: Critical
>              Labels: newbie, patch, test
>             Fix For: 2.0.4
>
>         Attachments: leakdetect.patch, screenshot-1.jpg, system.log
>
>
> Files cannot be deleted, patch CASSANDRA-5383 (Win7 deleting problem) doesn't help on
Win-7 on Cassandra 2.0.2. Even 2.1 Snapshot is not running. The cause is: Opened file handles
seem to be lost and not closed properly. Win 7 blames, that another process is still using
the file (but its obviously cassandra). Only restart of the server makes the files deleted.
But after heavy using (changes) of tables, there are about 24K files in the data folder (instead
of 35 after every restart) and Cassandra crashes. I experiminted and I found out, that a finalizer
fixes the problem. So after GC the files will be deleted (not optimal, but working fine).
It runs now 2 days continously without problem. Possible fix/test:
> I wrote the following finalizer at the end of class org.apache.cassandra.io.util.RandomAccessReader:
> {code:title=RandomAccessReader.java|borderStyle=solid}
> @Override
> protected void finalize() throws Throwable {
> 	deallocate();
> 	super.finalize();
> }
> {code}
> Can somebody test / develop / patch it? Thx.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message