incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Errors on replica nodes halt repair
Date Tue, 19 Mar 2013 08:01:02 GMT
It's easier to understand what's happening if you provide the full error message. 

It looks like out of order data in the files, nodetool scrub can fix that error. 

Try repairing a single CF at a time so you can work out which one is failing. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 19/03/2013, at 10:58 AM, Dane Miller <dane@optimalsocial.com> wrote:

> I'm having trouble completing a repair on several of my nodes due to
> errors during compaction.  This is a 6 node cluster using the simple
> replication strategy, rf=3, with each node assigned a single token.
> I'm running "nodetool repair -pr" on node1, which progresses until a
> specific keyspace then appears to hang.  On the replicas, nodes 2 and
> 3, I  find errors that seem related to compaction.  I'm considering
> running a scrub if I can determine the column family where the errors
> occur.  But I'm not confident I understand the problem, and I'm wary
> of making it worse. What's the best way to recover from these errors?
> Note, this cluster was recently upgraded from 1.1.6 to 1.2.1, then to
> 1.2.2.
> 
> 
> node2
> 
> ERROR [Thread-97275] 2013-03-13 23:51:30,359 CassandraDaemon.java
> (line 133) Exception in thread Thread[Thread-97275,5,main]
> java.lang.RuntimeException: Last written key
> DecoratedKey(161894077670705622023702574770140080251,
> 757365723a3a313a3a373537363636393130) >= current key DecoratedKey(
> 
> ERROR [CompactionExecutor:7697] 2013-03-15 21:45:59,584
> CassandraDaemon.java (line 133) Exception in thread Thread[Compactio
> nExecutor:7697,1,main]
> java.lang.AssertionError: originally calculated column size of
> 321455446 but now it is 321455483
> 
> 
> node3
> 
> ERROR [Thread-97525] 2013-03-13 23:51:44,788 CassandraDaemon.java
> (line 133) Exception in thread Thread[Thread-97525,5,main]
> java.lang.RuntimeException: Last written key
> DecoratedKey(161894077670705622023702574770140080251,
> 757365723a3a313a3a373537363636393130) >= current key DecoratedKey
> 
> ERROR [Thread-97564] 2013-03-13 23:54:03,403 CassandraDaemon.java
> (line 133) Exception in thread Thread[Thread-97564,5,main]
> java.lang.RuntimeException: Last written key
> DecoratedKey(152706250731373455824787766459206671594,
> 757365723a3a313a3a333434313038323239) >= current key DecoratedKey(
> 
> ERROR [Thread-661] 2013-03-15 21:02:05,981 CassandraDaemon.java (line
> 133) Exception in thread Thread[Thread-661,5,main]
> java.lang.NegativeArraySizeException
> 
> 
> Details:
> 6 node cluster
> cassandra  1.2.2 - single token per node
> RandomPartitioner, EC2Snitch
> Replication: SimpleStrategy, rf=3
> Ubuntu 10.10 x86_64
> EC2 m1.large
> 
> 
> Dane


Mime
View raw message