Hi Rob,

It was hardware.  Memory.  I've been loading data since I originally posted.  No exceptions so far.  I had some issues with OOMs when I first started playing with cassandra.  I increased the amount RAM to the VM and reduced the memtable size.  I'm guessing it's because I'm using I3s.  More cores would most likely improve GC performance.

I put all the logs and my configs on my google drive.  The link is in the original post.  I'm running 1.2.4.  There have been two releases since my original download.  I'm going to attempt an upgrade soon. 

I'm also considering using leveled compaction.  I just have two 750GB drives per node.  I'd like to use more than 50% of the drives if I can.

Thanks,
Glenn


On Mon, Jul 1, 2013 at 11:08 AM, Robert Coli <rcoli@eventbrite.com> wrote:
On Sat, Jun 29, 2013 at 8:39 PM, Glenn Thompson <gatman1001@gmail.com> wrote:
> I'm Glenn Thompson and new to Cassandra.  I have been trying to figure out
> how to recover from a CorruptBlockException.
> ...
> One of my nodes must have a hardware problem.  Although I've been unable to
> find anything wrong via logs, smart, or mce.
> ...
> The repair, scrub, and decommission all produced Exceptions related to the
> same few corrupt files.

Hardware problem sounds relatively likely, especially if you have not
crashed your nodes. Only other thing I can think of is an issue with
the relationship of the compression library and the JVM. What JVM/JDK
are you using, and what compression method is in use on the Column
Family?

In general the actions you took were reasonable. Do you have the full
stack trace?

=Rob