incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yuki Morishita <mor.y...@gmail.com>
Subject Re: Startup Exception During Upgrade 1.1.6 to 1.2.2 during LCS Migration and Corrupt Tables
Date Sat, 09 Mar 2013 05:14:32 GMT
Are you sure you are using 1.2.2?
Because LegacyLeveledManifest is from unreleased development version.


On Friday, March 8, 2013 at 11:02 PM, Arya Goudarzi wrote:

> Hi,
> 
> I am exercising the rolling upgrade from 1.1.6 to 1.2.2. When I upgraded to 1.2.2 on
the first node, during startup I got this exception:
> 
> ERROR [main] 2013-03-09 04:24:30,771 CassandraDaemon.java (line 213) Could not migrate
old leveled manifest. Move away the .json file in the data directory
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.cassandra.utils.EstimatedHistogram$EstimatedHistogramSerializer.deserialize(EstimatedHistogram.java:265)
> at org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:365)
> at org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:351)
> at org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:100)
> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:209)
> at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:391)
> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:434)  
> 
> This is when it is trying to migrate LCS I believe. I removed the Json files from data
directories:
> 
> data/ $ for i in `find . | grep 'json' | awk '{print $1}'`; do rm -rf $i; done 
> 
> Then during the second attempt at restart, I got the following exception:
> 
> ERROR [main] 2013-03-09 04:24:30,771 CassandraDaemon.java (line 213) Could not migrate
old leveled manifest. Move away the .json file in the data directory
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.cassandra.utils.EstimatedHistogram$EstimatedHistogramSerializer.deserialize(EstimatedHistogram.java:265)
> at org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:365)
> at org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:351)
> at org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:100)
> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:209)
> at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:391)
> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:434)
> 
> OK. I seems it created snapshots prior to migration step. So it is safe to remove those,
right? 
> 
> data/ $ for i in `find . | grep 'pre-sstablemetamigration' | awk '{print $1}'`; do rm
-rf $i; done  
> 
> Now startup again, but I see bunch of corrupt sstable logs messages: 
> 
> ERROR [SSTableBatchOpen:1] 2013-03-09 04:55:39,826 SSTableReader.java (line 242) Corrupt
sstable /var/lib/cassandra/data/keyspace_production/UniqueIndexes/keyspace_production-UniqueIndexes-hf-98318=[Filter.db,
Data.db, CompressionInfo.db, Statistics.db, Index.db]; skipped 
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.cassandra.utils.BloomFilterSerializer.deserialize(BloomFilterSerializer.java:45)
> at org.apache.cassandra.utils.Murmur2BloomFilter$Murmur2BloomFilterSerializer.deserialize(Murmur2BloomFilter.java:40)
> at org.apache.cassandra.utils.FilterFactory.deserialize(FilterFactory.java:71)
> at org.apache.cassandra.io.sstable.SSTableReader.loadBloomFilter(SSTableReader.java:334)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:199)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:149)
> at org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:238)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
> 
> This is worrisome. How should I deal with this situation? scrub maybe? Should I open
a bug?
> 
> Cheers,
> Arya
> 
> 
> 



Mime
View raw message