cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From André Cruz (JIRA) <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-2598) incremental_backups and snapshot_before_compaction duplicate hard links
Date Tue, 08 May 2012 11:09:49 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-2598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270362#comment-13270362
] 

André Cruz commented on CASSANDRA-2598:
---------------------------------------

I think I have a similar issue but I don't have incremental_backups or snapshot_before_compaction
enabled, and I'm using 1.1.

Since I upgraded to Cassandra 1.1, I get the following error when trying to delete a CF. After
this happens the CF is not accessible anymore, but I cannot create another one with the same
name until I restart the server.

INFO [MigrationStage:1] 2012-05-07 18:10:12,682 ColumnFamilyStore.java (line 634) Enqueuing
flush of Memtable-schema_columnfamilies@1128094887(978/1222 serialized/live bytes, 21 ops)
INFO [FlushWriter:2] 2012-05-07 18:10:12,682 Memtable.java (line 266) Writing Memtable-schema_columnfamilies@1128094887(978/1222
serialized/live bytes, 21 ops)
INFO [FlushWriter:2] 2012-05-07 18:10:12,720 Memtable.java (line 307) Completed flushing /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-28-Data.db
(1041 bytes)
INFO [MigrationStage:1] 2012-05-07 18:10:12,721 ColumnFamilyStore.java (line 634) Enqueuing
flush of Memtable-schema_columns@1599271050(392/490 serialized/live bytes, 8 ops)
INFO [FlushWriter:2] 2012-05-07 18:10:12,722 Memtable.java (line 266) Writing Memtable-schema_columns@1599271050(392/490
serialized/live bytes, 8 ops)
INFO [CompactionExecutor:8] 2012-05-07 18:10:12,722 CompactionTask.java (line 114) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-26-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-28-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfam
ilies-hc-27-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-25-Data.db')]
INFO [FlushWriter:2] 2012-05-07 18:10:12,806 Memtable.java (line 307) Completed flushing /var/lib/cassandra/data/system/schema_columns/system-schema_columns-hc-23-Data.db
(447 bytes)
INFO [CompactionExecutor:8] 2012-05-07 18:10:12,811 CompactionTask.java (line 225) Compacted
to [/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-29-Data.db,].
 24,797 to 21,431
(~86% of original) bytes for 2 keys at 0.232252MB/s.  Time: 88ms.
ERROR [MigrationStage:1] 2012-05-07 18:10:12,895 CLibrary.java (line 158) Unable to create
hard link
com.sun.jna.LastErrorException: errno was 17
at org.apache.cassandra.utils.CLibrary.link(Native Method)
at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:150)
at org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512)
at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403)
at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
at org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR [Thrift:17] 2012-05-07 18:10:12,898 CustomTThreadPoolServer.java (line 204) Error occurred
during processing of message.
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOError: java.io.IOException:
Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json to /var/lib/cassandra/data/
Disco/Client/snapshots/1336410612893-Client/Client.json (errno 17)
at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:372)
at org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:191)
at org.apache.cassandra.service.MigrationManager.announceColumnFamilyDrop(MigrationManager.java:182)
at org.apache.cassandra.thrift.CassandraServer.system_drop_column_family(CassandraServer.java:948)
at org.apache.cassandra.thrift.Cassandra$Processor$system_drop_column_family.getResult(Cassandra.java:3348)
at org.apache.cassandra.thrift.Cassandra$Processor$system_drop_column_family.getResult(Cassandra.java:3336)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.util.concurrent.ExecutionException: java.io.IOError: java.io.IOException:
Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json to /var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json
(errno 17)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:368)
... 11 more
Caused by: java.io.IOError: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json
to /var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json (errno
17)
at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1454)
at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512)
at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403)
at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
at org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
... 3 more
Caused by: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json
to /var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json (errno
17)
at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:163)
at org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
... 10 more
ERROR [MigrationStage:1] 2012-05-07 18:10:12,899 AbstractCassandraDaemon.java (line 134) Exception
in thread Thread[MigrationStage:1,5,main]
java.io.IOError: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json
to /var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json (errno
17)
at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1454)
at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512)
at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403)
at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
at org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/Disco/Client/Client.json
to /var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json (errno
17)
at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:163)
at org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
... 10 more


I've tried recreating the data dirs, to see if this was some kind of permissions problem,
but the error happens every time and with any CF. I'm using the Cassandra Debian package on
Debian squeeze and the Sun JVM (build 1.6.0_26-b03).
                
> incremental_backups and snapshot_before_compaction duplicate hard links
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-2598
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2598
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.8 beta 1
>         Environment: linux & jna
>            Reporter: Mck SembWever
>            Assignee: Jonathan Ellis
>            Priority: Minor
>             Fix For: 0.8.0
>
>         Attachments: 2598.txt
>
>
> See discussion @ http://thread.gmane.org/gmane.comp.db.cassandra.user/15933/
> Enabling both incremental_backups and snapshot_before_compaction leads to the same hard
links trying to be created.
> This gives stacktraces like 
> java.io.IOError: java.io.IOException: Unable to create hard link from
> /cassandra-data/<keyspace>/<cf>-f-3875-Data.db
> to
> /cassandra-data/<keyspace>/snapshots/compact-<cf>/<cf>-f-3875-Data.db
> (errno 17)
> 	at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1629)
> 	at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1654)
> 	at org.apache.cassandra.db.Table.snapshot(Table.java:198)
> 	at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:504)
> 	at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:146)
> 	at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:112)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: Unable to create hard link from
> /cassandra-data/<keyspace>/<cf>-f-3875-Data.db
> to
> /cassandra-data/<keyspace>/snapshots/compact-<cf>/<cf>-f-3875-Data.db
> (errno 17)
> 	at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:155)
> 	at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:713)
> 	at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1622)
> 	... 10 more

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message