Hi,

 

Just had a go at upgrading a node to the latest stable c* 2 release and think I ran into some issues with manifest migration.

 

On initial start up I hit this error as it starts to load the first of my CF.

 

INFO [main] 2013-09-24 22:56:01,018 LegacyLeveledManifest.java (line 89) Migrating manifest for struqrealtime/impressionstorev2

INFO [main] 2013-09-24 22:56:01,019 LegacyLeveledManifest.java (line 119) Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration

ERROR [main] 2013-09-24 22:56:01,030 CassandraDaemon.java (line 459) Exception encountered during startup

FSWriteError in /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json

        at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:83)

        at org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:138)

        at org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91)

        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)

        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)

        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)

Caused by: java.nio.file.NoSuchFileException: /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json -> /disk1/cassandra/data/struqrealtime/impressionstorev2/impressionstorev2.json

        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)

        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)

        at sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:474)

        at java.nio.file.Files.createLink(Files.java:1037)

        at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:79)

        ... 5 more

 

I had already successful run a test migration on our dev server. Only real difference I can see if the number of data directories defined and the amount of data being held.

 

I’ve run upgradesstables under 1.2.10. I have always been using vnodes and CQL3. I recently moved to using LZ4 instead of Snappy..

 

I tried to startup again and it gave me a slightly different error

 

INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 89) Migrating manifest for struqrealtime/impressionstorev2

INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 119) Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration

ERROR [main] 2013-09-24 22:58:28,222 CassandraDaemon.java (line 459) Exception encountered during startup

java.lang.RuntimeException: Tried to create duplicate hard link to /disk3/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/struqrealtime-impressionstorev2-ic-1030-TOC.txt

        at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)

        at org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:129)

        at org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91)

        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)

        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)

        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)

 

Will have a go recreating this tomorrow.

 

Any insight or guesses at what the issue might be are always welcome.

 

Thanks,

Chris