incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: CorruptSSTableException in system keyspace
Date Thu, 04 Apr 2013 00:47:53 GMT
There is a ticket there from an older version but I doubt thats it https://issues.apache.org/jira/browse/CASSANDRA-4837

You may be hitting an edge case by quickly creating 200 to 300 CF's. Can you reproduce the
problem outside of your test infrastructure ? If so can you raise a ticket https://issues.apache.org/jira/browse/CASSANDRA

Cheers

-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 2/04/2013, at 6:39 PM, Alexander Shutyaev <shutyaev@gmail.com> wrote:

> Hi, Aaron!
> 
> We were using 1.2.2. Now after your suggestion I've upgraded it to 1.2.3. That CorruptSSTableException
is now gone, but the problem is still here - after some time we start getting another exception
in cassandra logs (and transport exception on the client side). The new exception is java.lang.IllegalStateException:
One row required, 0 found. Full stacktrace available here [1]
> 
> [1] http://pastebin.com/W0nL6hK7
> 
> Thanks in advance,
> Alexander
> 
> 
> 2013/4/2 aaron morton <aaron@thelastpickle.com>
> What version are you using ?
> 
> There are two tickets with similar issues fixed in 1.2.X releases.
> 
> https://issues.apache.org/jira/browse/CASSANDRA-5225
> https://issues.apache.org/jira/browse/CASSANDRA-5365
> 
> Cheers
> 
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
> 
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 1/04/2013, at 1:45 PM, Alexander Shutyaev <shutyaev@gmail.com> wrote:
> 
> > Hi all!
> >
> > Our app currently moves from hibernate to cassandra+solr combination (we use our
own storage abstraction for this). We have a lot of junit integration tests that use database.
Now we've decided to use these tests to test our cassandra+solr storage implementation (given
that on hibernate implementation these tests run fine). For that purpose we've installed cassandra
and solr on a dedicated vm and we've launched the tests.
> >
> > Now there are two things that should be mentioned about these tests and how we use
cassandra.
> >
> > 1. We have 1 keyspace and about 200-300 column families.
> > 2. Tests are written in such a manner that at the beginning of each test we drop
the keyspace and then recreate it from scratch (together with all column families).
> > 3. On client side we use hector.
> >
> > The problem is that each time we launch the tests they start failing randomly at
some point. In cassandra logs we see a lot of CorruptSSTableException-s in system keyspace
caused by CorruptBlockException-s. An example of full stack trace of such exception can be
found here [1]. The tests start to fail due to thrift transport exceptions. At that point
we are also unable to connect to cassandra using cassandra-cli. Then, after some time cassandra
goes back to normal state all by itself.
> >
> > [1] http://pastebin.com/9SpEJpap
> >
> > P.S. I can provide full system.log and output.log from clean start (all data folders
erased) till the errors and later when the system is ok once again. Although I'm not sure
which file sharing service to utilise.
> >
> > Thanks in advance,
> > Alexander
> 
> 


Mime
View raw message