What version of Cassandra are you running?  It sounds like the disk failure policy is incorrectly being applied to scrub, which kind of defeats the purpose of it.  I'd recommend opening a ticket (https://issues.apache.org/jira/browse/CASSANDRA) with the information you posted here.

On Fri, Oct 2, 2015 at 3:01 AM, George Sigletos <sigletos@textkernel.nl> wrote:
I'm also facing problems regarding corrupt sstables and also couldn't run sstablescrub successfully.

I restarted my nodes with disk failure policy "best_effort", then I run the "nodetool scrub <keyspace>"

Once done I removed the corrupt tables manually and started repair


On Thu, Oct 1, 2015 at 7:27 PM, John Anderson <sontek@gmail.com> wrote:
I have a 25 node cluster and we lost power on one of the racks last night and now 6 of our nodes will not start up and we are getting the following error:

INFO  [main] 2015-10-01 10:19:22,111 CassandraDaemon.java:122 - JMX is enabled to receive remote connections on port: 7199
INFO  [main] 2015-10-01 10:19:22,124 CacheService.java:111 - Initializing key cache with capacity of 100 MBs.
INFO  [main] 2015-10-01 10:19:22,129 CacheService.java:133 - Initializing row cache with capacity of 0 MBs
INFO  [main] 2015-10-01 10:19:22,133 CacheService.java:150 - Initializing counter cache with capacity of 50 MBs
INFO  [main] 2015-10-01 10:19:22,135 CacheService.java:161 - Scheduling counter cache save to every 7200 seconds (going to save all keys).
INFO  [main] 2015-10-01 10:19:22,211 ColumnFamilyStore.java:363 - Initializing system.sstable_activity
INFO  [SSTableBatchOpen:1] 2015-10-01 10:19:22,639 SSTableReader.java:478 - Opening /mnt/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-45 (805 bytes)
ERROR [SSTableBatchOpen:1] 2015-10-01 10:19:22,657 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop"
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException
        at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:131) ~[apache-cassandra-2.1.9.jar:2.1.9]
        at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85) ~[apache-cassandra-2.1.9.jar:2.1.9]
        at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79) ~[apache-cassandra-2.1.9.jar:2.1.9]
        at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72) ~[apache-cassandra-2.1.9.jar:2.1.9]
        at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168) ~[apache-cassandra-2.1.9.jar:2.1.9]
        at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752) ~[apache-cassandra-2.1.9.jar:2.1.9]
        at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703) ~[apache-cassandra-2.1.9.jar:2.1.9]
        at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491) ~[apache-cassandra-2.1.9.jar:2.1.9]
        at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:387) ~[apache-cassandra-2.1.9.jar:2.1.9]
        at org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:534) ~[apache-cassandra-2.1.9.jar:2.1.9]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_60]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_60]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_60]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
Caused by: java.io.EOFException: null
        at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340) ~[na:1.8.0_60]
        at java.io.DataInputStream.readUTF(DataInputStream.java:589) ~[na:1.8.0_60]
        at java.io.DataInputStream.readUTF(DataInputStream.java:564) ~[na:1.8.0_60]
        at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:106) ~[apache-cassandra-2.1.9.jar:2.1.9]
        ... 14 common frames omitted


I found some people recommending scrubbing the sstable so I attempted that and got the following error:

bin/sstablescrub system sstable_activity -v                                                                                                                                                                  
ERROR 17:26:03 Exiting forcefully due to file system exception on startup, disk failure policy "stop"
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException
        at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:131) ~[apache-cassandra-2.1.9.jar:2.1.9]
        at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85) ~[apache-cassandra-2.1.9.jar:2.1.9]
        at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79) ~[apache-cassandra-2.1.9.jar:2.1.9]
        at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72) ~[apache-cassandra-2.1.9.jar:2.1.9]
        at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168) ~[apache-cassandra-2.1.9.jar:2.1.9]
        at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752) ~[apache-cassandra-2.1.9.jar:2.1.9]
        at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703) ~[apache-cassandra-2.1.9.jar:2.1.9]
        at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491) ~[apache-cassandra-2.1.9.jar:2.1.9]
        at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:387) ~[apache-cassandra-2.1.9.jar:2.1.9]
        at org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:534) ~[apache-cassandra-2.1.9.jar:2.1.9]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_60]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_60]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_60]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
Caused by: java.io.EOFException: null
        at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340) ~[na:1.8.0_60]
        at java.io.DataInputStream.readUTF(DataInputStream.java:589) ~[na:1.8.0_60]
        at java.io.DataInputStream.readUTF(DataInputStream.java:564) ~[na:1.8.0_60]
        at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:106) ~[apache-cassandra-2.1.9.jar:2.1.9]
        ... 14 common frames omitted


Is there a fix for this?




--
Tyler Hobbs
DataStax