cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nate McCall <n...@thelastpickle.com>
Subject Re: Currupt sstables when upgrading from 2.1.8 to 2.1.9
Date Tue, 15 Sep 2015 16:42:17 GMT
You have a/some corrupt SSTables. 2.1.9 is doing strict checking at startup
and reacting based on "disk_failure_policy" per the stack trace.

For details, see:
https://issues.apache.org/jira/browse/CASSANDRA-9686

Either way, you are going to have to run nodetool scrub. I'm not sure if
it's better to do this from 2.1.8 or from 2.1.9 with "disk_failure_policy:
ignore"

It feels like that option got overloaded a bit strangely with the changes
in CASSANDRA-9686 and I have not yet tried it with it's new meaning.

On Tue, Sep 15, 2015 at 5:26 AM, George Sigletos <sigletos@textkernel.nl>
wrote:

> Hello,
>
> I tried to upgrade two of our clusters from 2.1.8 to 2.1.9. In some, but
> not all nodes, I got errors about corrupt sstables when restarting. I
> downgraded back to 2.1.8 for now.
>
> Has anybody else faced the same problem? Should sstablescrub fix the
> problem? I ddin't tried that yet.
>
> Kind regards,
> George
>
> ERROR [SSTableBatchOpen:3] 2015-09-14 10:16:03,296 FileUtils.java:447 -
> Exiting forcefully due to file system exception on startup, disk failure
> policy "stop"
> org.apache.cassandra.io.sstable.CorruptSSTableException:
> java.io.EOFException
> at
> org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:131)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
>         at
> org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
>         at
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
>         at
> org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
>         at
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
>         at
> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
>         at
> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
>         at
> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
>         at
> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:387)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
>         at
> org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:534)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
>         at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source) [na:1.7.0_75]
>         at java.util.concurrent.FutureTask.run(Unknown Source)
> [na:1.7.0_75]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source) [na:1.7.0_75]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source) [na:1.7.0_75]
>         at java.lang.Thread.run(Unknown Source) [na:1.7.0_75]
> Caused by: java.io.EOFException: null
>         at java.io.DataInputStream.readUnsignedShort(Unknown Source)
> ~[na:1.7.0_75]
>         at java.io.DataInputStream.readUTF(Unknown Source) ~[na:1.7.0_75]
>         at java.io.DataInputStream.readUTF(Unknown Source) ~[na:1.7.0_75]
>         at
> org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:106)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
>



-- 
-----------------
Nate McCall
Austin, TX
@zznate

Co-Founder & Sr. Technical Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Mime
View raw message