incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcus Eriksson <krum...@gmail.com>
Subject Re: 1.2.10 -> 2.0.1 migration issue
Date Wed, 25 Sep 2013 13:09:24 GMT
you probably have to remove the old snapshots before trying to restart


On Wed, Sep 25, 2013 at 3:05 PM, Christopher Wirt <chris.wirt@struq.com>wrote:

> Should also say. I have managed to move one node from 1.2.10 to 2.0.0. I’m
> seeing this error on the machine I tried to migrate earlier to 2.0.1****
>
> ** **
>
> Thanks****
>
> ** **
>
> *From:* Christopher Wirt [mailto:chris.wirt@struq.com]
> *Sent:* 25 September 2013 14:04
> *To:* 'user@cassandra.apache.org'
> *Subject:* RE: 1.2.10 -> 2.0.1 migration issue****
>
> ** **
>
> Hi Marcus,****
>
> ** **
>
> I’ve seen your patch. This works with what I’m seeing. The first data
> directory only contained the JSON manifest at that time.****
>
> ** **
>
> As a workaround I’ve made sure that each of the snapshot directories now
> exist before starting up.****
>
> ** **
>
> I still end up with the second exception I posted regarding a duplicate
> hard link. Possibly two unrelated exceptions.****
>
> ** **
>
> After getting this error. Looking at the datadirs****
>
> Data1 contains ****
>
> JSON manifests****
>
> Loads of data files****
>
> Snapshot directory****
>
> Data2 contains****
>
>                 Just the snapshot directory****
>
> Data3 contains****
>
>                 Just the snapshot directory****
>
> ** **
>
> INFO 12:56:22,766 Migrating manifest for struqrealtime/impressionstorev2**
> **
>
> INFO 12:56:22,767 Snapshotting struqrealtime, impressionstorev2 to
> pre-sstablemetamigration****
>
> ERROR 12:56:22,787 Exception encountered during startup****
>
> java.lang.RuntimeException: Tried to create duplicate hard link to
> /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json
> ****
>
>         at
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)**
> **
>
>         at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:138)
> ****
>
>         at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91)
> ****
>
>         at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247)
> ****
>
>         at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443)
> ****
>
>         at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)
> ****
>
> java.lang.RuntimeException: Tried to create duplicate hard link to
> /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json
> ****
>
>         at
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)**
> **
>
>         at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:138)
> ****
>
>         at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91)
> ****
>
>         at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247)
> ****
>
>         at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443)
> ****
>
>         at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)
> ****
>
> Exception encountered during startup: Tried to create duplicate hard link
> to
> /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json
> ****
>
> ** **
>
> Thanks,****
>
> ** **
>
> Chris****
>
> ** **
>
> ** **
>
> *From:* Marcus Eriksson [mailto:krummas@gmail.com]
> *Sent:* 25 September 2013 13:11
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: 1.2.10 -> 2.0.1 migration issue****
>
> ** **
>
> you are probably reading trunk NEWS.txt****
>
> ** **
>
> read the ticket for explanation of what the issue was (it is a proper bug)
> ****
>
> ** **
>
> On Wed, Sep 25, 2013 at 12:59 PM, Christopher Wirt <chris.wirt@struq.com>
> wrote:****
>
> Hi Marcus,****
>
> Thanks for having a look at this.****
>
>  ****
>
> Just noticed this in the NEWS.txt ****
>
>  ****
>
> For *leveled* compaction users, 2.0 must be *atleast* started before****
>
>      upgrading to 2.1 due to the fact that the old JSON *leveled*****
>
>      manifest is migrated into the *sstable* *metadata* files on startup**
> **
>
>      in 2.0 and this code is gone from 2.1.****
>
>  ****
>
> Basically, my fault for skimming over this too quickly. ****
>
>  ****
>
> We will move from 1.2.10 -> 2.0 -> 2.1****
>
>  ****
>
> Thanks,****
>
> Chris****
>
>  ****
>
>  ****
>
> *From:* Marcus Eriksson [mailto:krummas@gmail.com]
> *Sent:* 25 September 2013 09:37
> *To:* user@cassandra.apache.org
> *Subject:* Re: 1.2.10 -> 2.0.1 migration issue****
>
>  ****
>
> cant really reproduce, could you update the ticket with a bit more info
> about your setup?****
>
>  ****
>
> do you have multiple .json files in your data dirs?****
>
>  ****
>
> On Wed, Sep 25, 2013 at 10:07 AM, Marcus Eriksson <krummas@gmail.com>
> wrote:****
>
> this is most likely a bug, filed
> https://issues.apache.org/jira/browse/CASSANDRA-6093 and will try to have
> a look today.****
>
>  ****
>
> On Wed, Sep 25, 2013 at 1:48 AM, Christopher Wirt <chris.wirt@struq.com>
> wrote:****
>
> Hi,****
>
>  ****
>
> Just had a go at upgrading a node to the latest stable c* 2 release and
> think I ran into some issues with manifest migration.****
>
>  ****
>
> On initial start up I hit this error as it starts to load the first of my
> CF. ****
>
>  ****
>
> INFO [main] 2013-09-24 22:56:01,018 LegacyLeveledManifest.java (line 89)
> Migrating manifest for struqrealtime/impressionstorev2****
>
> INFO [main] 2013-09-24 22:56:01,019 LegacyLeveledManifest.java (line 119)
> Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration*
> ***
>
> ERROR [main] 2013-09-24 22:56:01,030 CassandraDaemon.java (line 459)
> Exception encountered during startup****
>
> FSWriteError in
> /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json
> ****
>
>         at
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:83)**
> **
>
>         at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:138)
> ****
>
>         at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91)
> ****
>
>         at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)
> ****
>
>         at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)
> ****
>
>         at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
> ****
>
> Caused by: java.nio.file.NoSuchFileException:
> /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json
> ->
> /disk1/cassandra/data/struqrealtime/impressionstorev2/impressionstorev2.json
> ****
>
>         at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)****
>
>         at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)****
>
>         at
> sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:474)
> ****
>
>         at java.nio.file.Files.createLink(Files.java:1037)****
>
>         at
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:79)**
> **
>
>         ... 5 more****
>
>  ****
>
> I had already successful run a test migration on our dev server. Only real
> difference I can see if the number of data directories defined and the
> amount of data being held. ****
>
>  ****
>
> I’ve run upgradesstables under 1.2.10. I have always been using vnodes and
> CQL3. I recently moved to using LZ4 instead of Snappy..****
>
>  ****
>
> I tried to startup again and it gave me a slightly different error****
>
>  ****
>
> INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 89)
> Migrating manifest for struqrealtime/impressionstorev2****
>
> INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 119)
> Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration*
> ***
>
> ERROR [main] 2013-09-24 22:58:28,222 CassandraDaemon.java (line 459)
> Exception encountered during startup****
>
> java.lang.RuntimeException: Tried to create duplicate hard link to
> /disk3/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/struqrealtime-impressionstorev2-ic-1030-TOC.txt
> ****
>
>         at
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)**
> **
>
>         at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:129)
> ****
>
>         at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91)
> ****
>
>         at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)
> ****
>
>         at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)
> ****
>
>         at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
> ****
>
>  ****
>
> Will have a go recreating this tomorrow.****
>
>  ****
>
> Any insight or guesses at what the issue might be are always welcome.****
>
>  ****
>
> Thanks,****
>
> Chris****
>
>  ****
>
>  ****
>
> ** **
>

Mime
View raw message