incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Shutyaev <shuty...@gmail.com>
Subject CorruptSSTableException in system keyspace
Date Mon, 01 Apr 2013 08:15:49 GMT
Hi all!

Our app currently moves from *hibernate* to *cassandra+solr* combination
(we use our own storage abstraction for this). We have a lot of junit
integration tests that use database. Now we've decided to use these tests
to test our *cassandra+solr* storage implementation (given that on *
hibernate* implementation these tests run fine). For that purpose we've
installed *cassandra* and *solr* on a dedicated vm and we've launched the
tests.

Now there are two things that should be mentioned about these tests and how
we use cassandra.

1. We have 1 *keyspace* and about 200-300 *column families*.
2. Tests are written in such a manner that at the beginning of each test we
drop the *keyspace* and then recreate it from scratch (together with
all *column
families*).
3. On client side we use *hector*.

The problem is that each time we launch the tests they start failing
randomly at some point. In *cassandra* logs we see a lot of *
CorruptSSTableException*-s in *system keyspace *caused by *
CorruptBlockException*-s. An example of full stack trace of such exception
can be found here [1]. The tests start to fail due to thrift
*transport*exceptions. At that point we are also unable to connect to
*cassandra* using *cassandra-cli*. Then, after some time *cassandra* goes
back to normal state all by itself.

[1] http://pastebin.com/9SpEJpap

P.S. I can provide full system.log and output.log from clean start (all
data folders erased) till the errors and later when the system is ok once
again. Although I'm not sure which file sharing service to utilise.

Thanks in advance,
Alexander

Mime
View raw message