incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Glenn Thompson <>
Subject CorruptBlockException
Date Sun, 30 Jun 2013 03:39:22 GMT

I'm Glenn Thompson and new to Cassandra.  I have been trying to figure out
how to recover from a CorruptBlockException.  My travels have led me to
numerous email threads and trouble tickets.  I think I did the right
thing(s) based on my research.

My basic situation.

I'm running on non enterprise hardware.  My personal cloud playground.  8
identical mini-itx Gigabyte GA-Z77N systems running centos 6.4 84 bit.
 Each has 16GB of Ram and two 750GB wd black laptop drives(raid 0) and
i3-3220 processors(2 cores 4 threads).  Anyone interested is the hardware
can go here :

I've been loading NOAA ISH data in an effort to learn/slash evaluate

One of my nodes must have a hardware problem.  Although I've been unable to
find anything wrong via logs, smart, or mce.

Cassandra discovered the error during a compaction.  My loading continued
so I let it finish.

Then I:

and finally decommissioned the node.

At no point did Cassandra declare any of the tokens as down or anything.
 Other than the Exceptions In the logs.  Cassandra was happy.

The repair, scrub, and decommission all produced Exceptions related to the
same few corrupt files.

I plumb this whole thing with SaltStack so I'm going to start over and
attempt another load with new RAM in the bad node.  I'll save my logs and
configs if anyone is interested.  I'll post them on my google drive if
anyone thinks it will be useful.


View raw message