incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dane Miller <d...@optimalsocial.com>
Subject Errors on replica nodes halt repair
Date Mon, 18 Mar 2013 21:58:11 GMT
I'm having trouble completing a repair on several of my nodes due to
errors during compaction.  This is a 6 node cluster using the simple
replication strategy, rf=3, with each node assigned a single token.
I'm running "nodetool repair -pr" on node1, which progresses until a
specific keyspace then appears to hang.  On the replicas, nodes 2 and
3, I  find errors that seem related to compaction.  I'm considering
running a scrub if I can determine the column family where the errors
occur.  But I'm not confident I understand the problem, and I'm wary
of making it worse. What's the best way to recover from these errors?
Note, this cluster was recently upgraded from 1.1.6 to 1.2.1, then to
1.2.2.


node2

ERROR [Thread-97275] 2013-03-13 23:51:30,359 CassandraDaemon.java
(line 133) Exception in thread Thread[Thread-97275,5,main]
java.lang.RuntimeException: Last written key
DecoratedKey(161894077670705622023702574770140080251,
757365723a3a313a3a373537363636393130) >= current key DecoratedKey(

ERROR [CompactionExecutor:7697] 2013-03-15 21:45:59,584
CassandraDaemon.java (line 133) Exception in thread Thread[Compactio
nExecutor:7697,1,main]
java.lang.AssertionError: originally calculated column size of
321455446 but now it is 321455483


node3

ERROR [Thread-97525] 2013-03-13 23:51:44,788 CassandraDaemon.java
(line 133) Exception in thread Thread[Thread-97525,5,main]
java.lang.RuntimeException: Last written key
DecoratedKey(161894077670705622023702574770140080251,
757365723a3a313a3a373537363636393130) >= current key DecoratedKey

ERROR [Thread-97564] 2013-03-13 23:54:03,403 CassandraDaemon.java
(line 133) Exception in thread Thread[Thread-97564,5,main]
java.lang.RuntimeException: Last written key
DecoratedKey(152706250731373455824787766459206671594,
757365723a3a313a3a333434313038323239) >= current key DecoratedKey(

ERROR [Thread-661] 2013-03-15 21:02:05,981 CassandraDaemon.java (line
133) Exception in thread Thread[Thread-661,5,main]
java.lang.NegativeArraySizeException


Details:
6 node cluster
cassandra  1.2.2 - single token per node
RandomPartitioner, EC2Snitch
Replication: SimpleStrategy, rf=3
Ubuntu 10.10 x86_64
EC2 m1.large


Dane

Mime
View raw message