incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Morton <aa...@thelastpickle.com>
Subject Re: Nodetool repair exceptions in Cassandra 2.0.2
Date Thu, 12 Dec 2013 02:51:42 GMT
> [2013-12-08 11:04:02,047] Repair session ff16c510-5ff7-11e3-97c0-5973cc397f8f for range
(1246984843639507027,1266616572749926276] failed with error org.apache.cassandra.exceptions.RepairException:
[repair #ff16c510-5ff7-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, (1246984843639507027,1266616572749926276]]
Validation failed in /10.x.x.48
the 10.x.x.48 node sent a tree response (merkle tree) to this node that did not contain the
tree. This node then killed the repair session. 

Look for log messages on 10.x.x.48 that correlate with the repair session ID above. They may
look like 

logger.error("Failed creating a merkle tree for " + desc + ", " + initiator + " (see log for
details)”);

or 

logger.info(String.format("[repair #%s] Sending completed merkle tree to %s for %s/%s", desc.sessionId,
initiator, desc.keyspace, desc.columnFamily));

Hope that helps. 

-----------------
Aaron Morton
New Zealand
@aaronmorton

Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

On 10/12/2013, at 12:57 pm, Laing, Michael <michael.laing@nytimes.com> wrote:

> My experience is that you must upgrade to 2.0.3 ASAP to fix this.
> 
> Michael
> 
> 
> On Mon, Dec 9, 2013 at 6:39 PM, David Laube <dave@stormpath.com> wrote:
> Hi All,
> 
> We are running Cassandra 2.0.2 and have recently stumbled upon an issue with nodetool
repair. Upon running nodetool repair on each of the 5 nodes in the ring (one at a time) we
observe the following exceptions returned to standard out;
> 
> 
> [2013-12-08 11:04:02,047] Repair session ff16c510-5ff7-11e3-97c0-5973cc397f8f for range
(1246984843639507027,1266616572749926276] failed with error org.apache.cassandra.exceptions.RepairException:
[repair #ff16c510-5ff7-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, (1246984843639507027,1266616572749926276]]
Validation failed in /10.x.x.48
> [2013-12-08 11:04:02,063] Repair session 284c8b40-5ff8-11e3-97c0-5973cc397f8f for range
(-109256956528331396,-89316884701275697] failed with error org.apache.cassandra.exceptions.RepairException:
[repair #284c8b40-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family2, (-109256956528331396,-89316884701275697]]
Validation failed in /10.x.x.103
> [2013-12-08 11:04:02,070] Repair session 399e7160-5ff8-11e3-97c0-5973cc397f8f for range
(8901153810410866970,8915879751739915956] failed with error org.apache.cassandra.exceptions.RepairException:
[repair #399e7160-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, (8901153810410866970,8915879751739915956]]
Validation failed in /10.x.x.103
> [2013-12-08 11:04:02,072] Repair session 3ea73340-5ff8-11e3-97c0-5973cc397f8f for range
(1149084504576970235,1190026362216198862] failed with error org.apache.cassandra.exceptions.RepairException:
[repair #3ea73340-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, (1149084504576970235,1190026362216198862]]
Validation failed in /10.x.x.103
> [2013-12-08 11:04:02,091] Repair session 6f0da460-5ff8-11e3-97c0-5973cc397f8f for range
(-5407189524618266750,-5389231566389960750] failed with error org.apache.cassandra.exceptions.RepairException:
[repair #6f0da460-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, (-5407189524618266750,-5389231566389960750]]
Validation failed in /10.x.x.103
> [2013-12-09 23:16:36,962] Repair session 7efc2740-6127-11e3-97c0-5973cc397f8f for range
(1246984843639507027,1266616572749926276] failed with error org.apache.cassandra.exceptions.RepairException:
[repair #7efc2740-6127-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, (1246984843639507027,1266616572749926276]]
Validation failed in /10.x.x.48
> [2013-12-09 23:16:36,986] Repair session a8c44260-6127-11e3-97c0-5973cc397f8f for range
(-109256956528331396,-89316884701275697] failed with error org.apache.cassandra.exceptions.RepairException:
[repair #a8c44260-6127-11e3-97c0-5973cc397f8f on keyspace_name/col_family2, (-109256956528331396,-89316884701275697]]
Validation failed in /10.x.x.210
> 
> The /var/log/cassandra/system.log shows similar info as above with no real explanation
as to the root cause behind the exception(s).  There also does not appear to be any additional
info in /var/log/cassandra/cassandra.log. We have tried restoring a recent snapshot of the
keyespace in question to a separate staging ring and the repair runs successfully and without
exception there. This is even after we tried insert/delete on the keyspace in the separate
staging ring. Has anyone seen this behavior before and what can we do to resolve this? Any
assistance would be greatly appreciated.
> 
> Best regards,
> -Dave
> 


Mime
View raw message