cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Philip Thompson (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-9133) Getting validationexception while repairing nodes
Date Wed, 08 Apr 2015 02:13:12 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-9133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Philip Thompson updated CASSANDRA-9133:
---------------------------------------
    Description: 
Cassandra V:2.01.4

We are running repairs on nodes to run scrub on them later to get around issue (CASSANDRA-9126).
While running nodetool repair on nodes, getting the validationexception error. We are seeing
bunch of these validation exceptions on lot of nodes. Any help is appreciated? 
{code}
java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException: [repair #b3ee6050-dd7a-11e4-adac-172f3292c10d
on ...., (-1729382256910270568,-1537228672809129408]] Validation failed in /........
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.cassandra.exceptions.RepairException: [repair #b3ee6050-dd7a-11e4-adac-172f3292c10d
on ...., (-1729382256910270568,-1537228672809129408]] Validation failed in .....
at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:161)
at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:228)
at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:59)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
{code}

Also, seeing the following exceptions:{code}
ERROR [ValidationExecutor:33] 2015-04-08 00:16:45,103 Validator.java (line 242) Failed creating
a merkle tree for [repair #bf8d13c0-dd7a-11e4-80ff-810270bd0450 on ......, (-8262604116349070008,-8070450532247928848]],
/.... (see log for details)
ERROR [ValidationExecutor:33] 2015-04-08 00:16:45,103 CassandraDaemon.java (line 258) Exception
in thread Thread[ValidationExecutor:33,1,main]
java.lang.AssertionError: row DecoratedKey(-8232281172744762721, 354036383831303736375f3330335f31355f474d4543454f)
received out of order wrt DecoratedKey(-8070630157486006754, 3540383834383134305f3336315f31355f474d4543454f)
at org.apache.cassandra.repair.Validator.add(Validator.java:136)
at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:806)
at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:63)
at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:404)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)	{code}

  was:
Cassandra V:2.01.4

We are running repairs on nodes to run scrub on them later to get around issue (CASSANDRA-9126).
While running nodetool repair on nodes, getting the validationexception error. We are seeing
bunch of these validation exceptions on lot of nodes. Any help is appreciated? 

java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException: [repair #b3ee6050-dd7a-11e4-adac-172f3292c10d
on ...., (-1729382256910270568,-1537228672809129408]] Validation failed in /........
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.cassandra.exceptions.RepairException: [repair #b3ee6050-dd7a-11e4-adac-172f3292c10d
on ...., (-1729382256910270568,-1537228672809129408]] Validation failed in .....
at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:161)
at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:228)
at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:59)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)


Also, seeing the following exceptions:
ERROR [ValidationExecutor:33] 2015-04-08 00:16:45,103 Validator.java (line 242) Failed creating
a merkle tree for [repair #bf8d13c0-dd7a-11e4-80ff-810270bd0450 on ......, (-8262604116349070008,-8070450532247928848]],
/.... (see log for details)
ERROR [ValidationExecutor:33] 2015-04-08 00:16:45,103 CassandraDaemon.java (line 258) Exception
in thread Thread[ValidationExecutor:33,1,main]
java.lang.AssertionError: row DecoratedKey(-8232281172744762721, 354036383831303736375f3330335f31355f474d4543454f)
received out of order wrt DecoratedKey(-8070630157486006754, 3540383834383134305f3336315f31355f474d4543454f)
at org.apache.cassandra.repair.Validator.add(Validator.java:136)
at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:806)
at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:63)
at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:404)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)	


> Getting validationexception while repairing nodes
> -------------------------------------------------
>
>                 Key: CASSANDRA-9133
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9133
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: srinivasu gottipati
>            Priority: Blocker
>             Fix For: 2.0.15
>
>
> Cassandra V:2.01.4
> We are running repairs on nodes to run scrub on them later to get around issue (CASSANDRA-9126).
While running nodetool repair on nodes, getting the validationexception error. We are seeing
bunch of these validation exceptions on lot of nodes. Any help is appreciated? 
> {code}
> java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException: [repair
#b3ee6050-dd7a-11e4-adac-172f3292c10d on ...., (-1729382256910270568,-1537228672809129408]]
Validation failed in /........
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.cassandra.exceptions.RepairException: [repair #b3ee6050-dd7a-11e4-adac-172f3292c10d
on ...., (-1729382256910270568,-1537228672809129408]] Validation failed in .....
> at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:161)
> at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:228)
> at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:59)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
> {code}
> Also, seeing the following exceptions:{code}
> ERROR [ValidationExecutor:33] 2015-04-08 00:16:45,103 Validator.java (line 242) Failed
creating a merkle tree for [repair #bf8d13c0-dd7a-11e4-80ff-810270bd0450 on ......, (-8262604116349070008,-8070450532247928848]],
/.... (see log for details)
> ERROR [ValidationExecutor:33] 2015-04-08 00:16:45,103 CassandraDaemon.java (line 258)
Exception in thread Thread[ValidationExecutor:33,1,main]
> java.lang.AssertionError: row DecoratedKey(-8232281172744762721, 354036383831303736375f3330335f31355f474d4543454f)
received out of order wrt DecoratedKey(-8070630157486006754, 3540383834383134305f3336315f31355f474d4543454f)
> at org.apache.cassandra.repair.Validator.add(Validator.java:136)
> at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:806)
> at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:63)
> at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:404)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)	{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message