cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Anderson <son...@gmail.com>
Subject Re: Repair corrupt SSTable from power outage?
Date Tue, 06 Oct 2015 16:56:45 GMT
I'm running 2.1.9

On Fri, Oct 2, 2015 at 8:40 AM, Tyler Hobbs <tyler@datastax.com> wrote:

> What version of Cassandra are you running?  It sounds like the disk
> failure policy is incorrectly being applied to scrub, which kind of defeats
> the purpose of it.  I'd recommend opening a ticket (
> https://issues.apache.org/jira/browse/CASSANDRA) with the information you
> posted here.
>
> On Fri, Oct 2, 2015 at 3:01 AM, George Sigletos <sigletos@textkernel.nl>
> wrote:
>
>> I'm also facing problems regarding corrupt sstables and also couldn't run
>> sstablescrub successfully.
>>
>> I restarted my nodes with disk failure policy "best_effort", then I run
>> the "nodetool scrub <keyspace>"
>>
>> Once done I removed the corrupt tables manually and started repair
>>
>>
>> On Thu, Oct 1, 2015 at 7:27 PM, John Anderson <sontek@gmail.com> wrote:
>>
>>> I have a 25 node cluster and we lost power on one of the racks last
>>> night and now 6 of our nodes will not start up and we are getting the
>>> following error:
>>>
>>> INFO  [main] 2015-10-01 10:19:22,111 CassandraDaemon.java:122 - JMX is
>>> enabled to receive remote connections on port: 7199
>>> INFO  [main] 2015-10-01 10:19:22,124 CacheService.java:111 -
>>> Initializing key cache with capacity of 100 MBs.
>>> INFO  [main] 2015-10-01 10:19:22,129 CacheService.java:133 -
>>> Initializing row cache with capacity of 0 MBs
>>> INFO  [main] 2015-10-01 10:19:22,133 CacheService.java:150 -
>>> Initializing counter cache with capacity of 50 MBs
>>> INFO  [main] 2015-10-01 10:19:22,135 CacheService.java:161 - Scheduling
>>> counter cache save to every 7200 seconds (going to save all keys).
>>> INFO  [main] 2015-10-01 10:19:22,211 ColumnFamilyStore.java:363 -
>>> Initializing system.sstable_activity
>>> INFO  [SSTableBatchOpen:1] 2015-10-01 10:19:22,639
>>> SSTableReader.java:478 - Opening
>>> /mnt/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-45
>>> (805 bytes)
>>> ERROR [SSTableBatchOpen:1] 2015-10-01 10:19:22,657 FileUtils.java:447 -
>>> Exiting forcefully due to file system exception on startup, disk failure
>>> policy "stop"
>>> org.apache.cassandra.io.sstable.CorruptSSTableException:
>>> java.io.EOFException
>>>         at
>>> org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:131)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>>         at
>>> org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>>         at
>>> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>>         at
>>> org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>>         at
>>> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>>         at
>>> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>>         at
>>> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>>         at
>>> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>>         at
>>> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:387)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>>         at
>>> org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:534)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>>         at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> [na:1.8.0_60]
>>>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> [na:1.8.0_60]
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> [na:1.8.0_60]
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> [na:1.8.0_60]
>>>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>>> Caused by: java.io.EOFException: null
>>>         at
>>> java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
>>> ~[na:1.8.0_60]
>>>         at java.io.DataInputStream.readUTF(DataInputStream.java:589)
>>> ~[na:1.8.0_60]
>>>         at java.io.DataInputStream.readUTF(DataInputStream.java:564)
>>> ~[na:1.8.0_60]
>>>         at
>>> org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:106)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>>         ... 14 common frames omitted
>>>
>>>
>>> I found some people recommending scrubbing the sstable so I attempted
>>> that and got the following error:
>>>
>>> bin/sstablescrub system sstable_activity -v
>>>
>>>
>>> ERROR 17:26:03 Exiting forcefully due to file system exception on
>>> startup, disk failure policy "stop"
>>> org.apache.cassandra.io.sstable.CorruptSSTableException:
>>> java.io.EOFException
>>>         at
>>> org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:131)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>>         at
>>> org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>>         at
>>> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>>         at
>>> org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>>         at
>>> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>>         at
>>> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>>         at
>>> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>>         at
>>> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>>         at
>>> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:387)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>>         at
>>> org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:534)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>>         at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> [na:1.8.0_60]
>>>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> [na:1.8.0_60]
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> [na:1.8.0_60]
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> [na:1.8.0_60]
>>>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>>> Caused by: java.io.EOFException: null
>>>         at
>>> java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
>>> ~[na:1.8.0_60]
>>>         at java.io.DataInputStream.readUTF(DataInputStream.java:589)
>>> ~[na:1.8.0_60]
>>>         at java.io.DataInputStream.readUTF(DataInputStream.java:564)
>>> ~[na:1.8.0_60]
>>>         at
>>> org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:106)
>>> ~[apache-cassandra-2.1.9.jar:2.1.9]
>>>         ... 14 common frames omitted
>>>
>>>
>>> Is there a fix for this?
>>>
>>
>>
>
>
> --
> Tyler Hobbs
> DataStax <http://datastax.com/>
>

Mime
View raw message