incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christopher Wirt" <chris.w...@struq.com>
Subject RE: 1.2.10 -> 2.0.1 migration issue
Date Wed, 25 Sep 2013 13:03:45 GMT
Hi Marcus,

 

I've seen your patch. This works with what I'm seeing. The first data
directory only contained the JSON manifest at that time.

 

As a workaround I've made sure that each of the snapshot directories now
exist before starting up.

 

I still end up with the second exception I posted regarding a duplicate hard
link. Possibly two unrelated exceptions.

 

After getting this error. Looking at the datadirs

Data1 contains 

JSON manifests

Loads of data files

Snapshot directory

Data2 contains

                Just the snapshot directory

Data3 contains

                Just the snapshot directory

 

INFO 12:56:22,766 Migrating manifest for struqrealtime/impressionstorev2

INFO 12:56:22,767 Snapshotting struqrealtime, impressionstorev2 to
pre-sstablemetamigration

ERROR 12:56:22,787 Exception encountered during startup

java.lang.RuntimeException: Tried to create duplicate hard link to
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json

        at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)

        at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(
LegacyLeveledManifest.java:138)

        at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le
gacyLeveledManifest.java:91)

        at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247)

        at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4
43)

        at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)

java.lang.RuntimeException: Tried to create duplicate hard link to
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json

        at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)

        at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(
LegacyLeveledManifest.java:138)

        at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le
gacyLeveledManifest.java:91)

        at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247)

        at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4
43)

        at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)

Exception encountered during startup: Tried to create duplicate hard link to
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json

 

Thanks,

 

Chris

 

 

From: Marcus Eriksson [mailto:krummas@gmail.com] 
Sent: 25 September 2013 13:11
To: user@cassandra.apache.org
Subject: Re: 1.2.10 -> 2.0.1 migration issue

 

you are probably reading trunk NEWS.txt

 

read the ticket for explanation of what the issue was (it is a proper bug)

 

On Wed, Sep 25, 2013 at 12:59 PM, Christopher Wirt <chris.wirt@struq.com>
wrote:

Hi Marcus,

Thanks for having a look at this.

 

Just noticed this in the NEWS.txt 

 

For leveled compaction users, 2.0 must be atleast started before

     upgrading to 2.1 due to the fact that the old JSON leveled

     manifest is migrated into the sstable metadata files on startup

     in 2.0 and this code is gone from 2.1.

 

Basically, my fault for skimming over this too quickly. 

 

We will move from 1.2.10 -> 2.0 -> 2.1

 

Thanks,

Chris

 

 

From: Marcus Eriksson [mailto:krummas@gmail.com] 
Sent: 25 September 2013 09:37
To: user@cassandra.apache.org
Subject: Re: 1.2.10 -> 2.0.1 migration issue

 

cant really reproduce, could you update the ticket with a bit more info
about your setup?

 

do you have multiple .json files in your data dirs?

 

On Wed, Sep 25, 2013 at 10:07 AM, Marcus Eriksson <krummas@gmail.com> wrote:

this is most likely a bug, filed
https://issues.apache.org/jira/browse/CASSANDRA-6093 and will try to have a
look today.

 

On Wed, Sep 25, 2013 at 1:48 AM, Christopher Wirt <chris.wirt@struq.com>
wrote:

Hi,

 

Just had a go at upgrading a node to the latest stable c* 2 release and
think I ran into some issues with manifest migration.

 

On initial start up I hit this error as it starts to load the first of my
CF. 

 

INFO [main] 2013-09-24 22:56:01,018 LegacyLeveledManifest.java (line 89)
Migrating manifest for struqrealtime/impressionstorev2

INFO [main] 2013-09-24 22:56:01,019 LegacyLeveledManifest.java (line 119)
Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration

ERROR [main] 2013-09-24 22:56:01,030 CassandraDaemon.java (line 459)
Exception encountered during startup

FSWriteError in
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json

        at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:83)

        at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(
LegacyLeveledManifest.java:138)

        at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le
gacyLeveledManifest.java:91)

        at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)

        at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4
42)

        at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)

Caused by: java.nio.file.NoSuchFileException:
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json ->
/disk1/cassandra/data/struqrealtime/impressionstorev2/impressionstorev2.json

        at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)

        at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)

        at
sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:474
)

        at java.nio.file.Files.createLink(Files.java:1037)

        at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:79)

        ... 5 more

 

I had already successful run a test migration on our dev server. Only real
difference I can see if the number of data directories defined and the
amount of data being held. 

 

I've run upgradesstables under 1.2.10. I have always been using vnodes and
CQL3. I recently moved to using LZ4 instead of Snappy..

 

I tried to startup again and it gave me a slightly different error

 

INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 89)
Migrating manifest for struqrealtime/impressionstorev2

INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 119)
Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration

ERROR [main] 2013-09-24 22:58:28,222 CassandraDaemon.java (line 459)
Exception encountered during startup

java.lang.RuntimeException: Tried to create duplicate hard link to
/disk3/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/struqrealtime-impressionstorev2-ic-1030-TOC.txt

        at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)

        at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(
LegacyLeveledManifest.java:129)

        at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le
gacyLeveledManifest.java:91)

        at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)

        at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4
42)

        at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)

 

Will have a go recreating this tomorrow.

 

Any insight or guesses at what the issue might be are always welcome.

 

Thanks,

Chris

 

 

 


Mime
View raw message