cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andreas Schnitzerling (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-6283) Windows 7 data files keept open / can't be deleted after compaction.
Date Thu, 28 Nov 2013 11:57:36 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13830136#comment-13830136
] 

Andreas Schnitzerling edited comment on CASSANDRA-6283 at 11/28/13 11:56 AM:
-----------------------------------------------------------------------------

I deployed now 2.0.3-tentative on the whole cluster. Result: With nodetool repair patch for
CASSANDRA-6275 doen't work. Neighbour nodes crash again with disk_failure_policy=stop.
{panel:title=system.log}
ERROR [ValidationExecutor:3] 2013-11-22 18:21:49,591 FileUtils.java (line 417) Stopping gossiper
 WARN [ValidationExecutor:3] 2013-11-22 18:21:49,591 StorageService.java (line 279) Stopping
gossip by operator request
ERROR [ValidationExecutor:4] 2013-11-22 18:21:50,361 Validator.java (line 242) Failed creating
a merkle tree for [repair #923a7360-539a-11e3-8fde-eb1c24a59bb8 on nieste/evrangesdevice,
(-787066926799647148,-773294852829911898]], /10.9.9.240 (see log for details)
ERROR [ValidationExecutor:4] 2013-11-22 18:21:50,371 CassandraDaemon.java (line 187) Exception
in thread Thread[ValidationExecutor:4,1,main]
FSWriteError in D:\Programme\cassandra\data\nieste\evrangesdevice\snapshots\923a7360-539a-11e3-8fde-eb1c24a59bb8\nieste-evrangesdevice-jb-9-Index.db
	at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:120)
	at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:382)
	at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:378)
	at org.apache.cassandra.db.Directories.clearSnapshot(Directories.java:416)
	at org.apache.cassandra.db.ColumnFamilyStore.clearSnapshot(ColumnFamilyStore.java:1801)
	at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:810)
	at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:62)
	at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:397)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: java.nio.file.FileSystemException: D:\Programme\cassandra\data\nieste\evrangesdevice\snapshots\923a7360-539a-11e3-8fde-eb1c24a59bb8\nieste-evrangesdevice-jb-9-Index.db:
Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet
wird.

	at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
	at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
	at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
	at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
	at sun.nio.fs.AbstractFileSystemProvider.delete(Unknown Source)
	at java.nio.file.Files.delete(Unknown Source)
	at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:116)
	... 11 more
ERROR [ValidationExecutor:4] 2013-11-22 18:21:50,371 FileUtils.java (line 417) Stopping gossiper
 WARN [ValidationExecutor:4] 2013-11-22 18:21:50,371 StorageService.java (line 279) Stopping
gossip by operator request
ERROR [ValidationExecutor:2] 2013-11-22 18:21:51,221 FileUtils.java (line 423) Stopping RPC
server
ERROR [ValidationExecutor:2] 2013-11-22 18:21:51,221 FileUtils.java (line 429) Stopping native
transport
{panel}


was (Author: andie78):
I deployed now 2.0.3-tentative on the whole cluster. Result: With nodetool repair patch for
CASSANDRA-6275 doen't work. Neighbour nodes crash again with disk_failure_policy=stop.

ERROR [ValidationExecutor:3] 2013-11-22 18:21:49,591 FileUtils.java (line 417) Stopping gossiper
 WARN [ValidationExecutor:3] 2013-11-22 18:21:49,591 StorageService.java (line 279) Stopping
gossip by operator request
ERROR [ValidationExecutor:4] 2013-11-22 18:21:50,361 Validator.java (line 242) Failed creating
a merkle tree for [repair #923a7360-539a-11e3-8fde-eb1c24a59bb8 on nieste/evrangesdevice,
(-787066926799647148,-773294852829911898]], /10.9.9.240 (see log for details)
ERROR [ValidationExecutor:4] 2013-11-22 18:21:50,371 CassandraDaemon.java (line 187) Exception
in thread Thread[ValidationExecutor:4,1,main]
FSWriteError in D:\Programme\cassandra\data\nieste\evrangesdevice\snapshots\923a7360-539a-11e3-8fde-eb1c24a59bb8\nieste-evrangesdevice-jb-9-Index.db
	at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:120)
	at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:382)
	at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:378)
	at org.apache.cassandra.db.Directories.clearSnapshot(Directories.java:416)
	at org.apache.cassandra.db.ColumnFamilyStore.clearSnapshot(ColumnFamilyStore.java:1801)
	at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:810)
	at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:62)
	at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:397)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: java.nio.file.FileSystemException: D:\Programme\cassandra\data\nieste\evrangesdevice\snapshots\923a7360-539a-11e3-8fde-eb1c24a59bb8\nieste-evrangesdevice-jb-9-Index.db:
Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet
wird.

	at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
	at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
	at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
	at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
	at sun.nio.fs.AbstractFileSystemProvider.delete(Unknown Source)
	at java.nio.file.Files.delete(Unknown Source)
	at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:116)
	... 11 more
ERROR [ValidationExecutor:4] 2013-11-22 18:21:50,371 FileUtils.java (line 417) Stopping gossiper
 WARN [ValidationExecutor:4] 2013-11-22 18:21:50,371 StorageService.java (line 279) Stopping
gossip by operator request
ERROR [ValidationExecutor:2] 2013-11-22 18:21:51,221 FileUtils.java (line 423) Stopping RPC
server
ERROR [ValidationExecutor:2] 2013-11-22 18:21:51,221 FileUtils.java (line 429) Stopping native
transport


> Windows 7 data files keept open / can't be deleted after compaction.
> --------------------------------------------------------------------
>
>                 Key: CASSANDRA-6283
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6283
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Windows 7 (32) / Java 1.7.0.45
>            Reporter: Andreas Schnitzerling
>            Priority: Critical
>              Labels: newbie, patch, test
>             Fix For: 2.0.4
>
>         Attachments: leakdetect.patch, screenshot-1.jpg, system.log
>
>
> Files cannot be deleted, patch CASSANDRA-5383 (Win7 deleting problem) doesn't help on
Win-7 on Cassandra 2.0.2. Even 2.1 Snapshot is not running. The cause is: Opened file handles
seem to be lost and not closed properly. Win 7 blames, that another process is still using
the file (but its obviously cassandra). Only restart of the server makes the files deleted.
But after heavy using (changes) of tables, there are about 24K files in the data folder (instead
of 35 after every restart) and Cassandra crashes. I experiminted and I found out, that a finalizer
fixes the problem. So after GC the files will be deleted (not optimal, but working fine).
It runs now 2 days continously without problem. Possible fix/test:
> I wrote the following finalizer at the end of class org.apache.cassandra.io.util.RandomAccessReader:
> {code:title=RandomAccessReader.java|borderStyle=solid}
> @Override
> protected void finalize() throws Throwable {
> 	deallocate();
> 	super.finalize();
> }
> {code}
> Can somebody test / develop / patch it? Thx.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message