incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: RuntimeException during leveled compaction
Date Tue, 12 Feb 2013 17:14:36 GMT
snapshot all nodes so you have a backup: nodetool snapshot -t corrupt

run nodetool scrub on the errant CF. 

Look for messages such as:

"Out of order row detected…"
"%d out of order rows found while scrubbing %s; Those have been written (in order) to a new
sstable (%s)"

In the logs. 

Cheers
  
-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 12/02/2013, at 6:13 AM, Andre Sprenger <andre.sprenger@getanet.de> wrote:

> Hi,
> 
> I'm running a 6 node Cassandra 1.1.5 cluster on EC2. We have switched to leveled compaction
a couple of weeks ago, 
> this has been successful. Some days ago 3 of the nodes start to log the following exception
during compaction of 
> a particular column family:
> 
> ERROR [CompactionExecutor:726] 2013-02-11 13:02:26,582 AbstractCassandraDaemon.java (line
135) Exception in thread Thread[CompactionExecutor:726,1,main]
> java.lang.RuntimeException: Last written key DecoratedKey(84590743047470232854915142878708713938,
31333535333333383530323237303130313030303232313537303030303132393832) 
> >= current key DecoratedKey(28357704665244162161305918843747894551, 31333430313336313830333831303130313030303230313632303030303036363338)

> writing into /var/cassandra/data/AdServer/EventHistory/Adserver-EventHistory-tmp-he-68638-Data.db
>         at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:134)
>         at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:153)
>         at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
>         at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:50)
>         at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> 
> Compaction does not happen any more for the column family and read performance gets worse
because of the growing 
> number of data files accessed during reads. Looks like one or more of the data files
are corrupt and have keys
> that are stored out of order.
> 
> Any help to resolve this situation would be greatly appreciated.
> 
> Thanks
> Andre
> 


Mime
View raw message