cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yuki Morishita (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-8120) Failed creating a merkle tree during repair.
Date Tue, 14 Oct 2014 22:24:34 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-8120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14171640#comment-14171640
] 

Yuki Morishita commented on CASSANDRA-8120:
-------------------------------------------

What option did you used for repair?
And what is your compaction strategy on censored/censored?


> Failed creating a merkle tree during repair.
> --------------------------------------------
>
>                 Key: CASSANDRA-8120
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8120
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 12.04.5 LTS.  3.2.0-69-virtual.
>            Reporter: jason milliron
>
> Seeing this while running nodetool repair on several nodes of our new (not an upgrade)
2.1.0 cluster.  Multiple keyspaces.  Doesn't hang the repair job.  Storage is SSD ephemeral
(EC2).  No storage errors in syslog.  
> ERROR [ValidationExecutor:39] 2014-10-13 22:20:51,140 Validator.java:242 - Failed creating
a merkle tree for [repair #1523dfa0-5327-11e4-925c-65a001aa6d80 on censored/censored, (-4468426774357114003,-4433737571269222163]],
/10.37.666.175 (see log for details)
> INFO  [AntiEntropySessions:37] 2014-10-13 22:20:51,141 RepairSession.java:260 - [repair
#308e2750-5327-11e4-925c-65a001aa6d80] new session: will sync ip-10-37-666-175.ec2.internal/10.37.666.175,
/10.167.666.137, /10.164.666.81 on range (6870883176570587821,6871062738154464368] for censored.[censored]
> INFO  [AntiEntropyStage:1] 2014-10-13 22:20:51,142 ColumnFamilyStore.java:856 - Enqueuing
flush of censored: 3227760 (0%) on-heap, 0 (0%) off-heap
> INFO  [MemtableFlushWriter:2295] 2014-10-13 22:20:51,145 Memtable.java:326 - Writing
Memtable-censored@2052537453(380369 serialized bytes, 8235 ops, 0%/0% of on/off-heap limit)
> ERROR [AntiEntropySessions:34] 2014-10-13 22:20:51,148 RepairSession.java:303 - [repair
#1523dfa0-5327-11e4-925c-65a001aa6d80] session completed with the following error
> org.apache.cassandra.exceptions.RepairException: [repair #1523dfa0-5327-11e4-925c-65a001aa6d80
on censored/censored, (-4468426774357114003,-4433737571269222163]] Validation failed in /10.37.666.175
>         at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:166)
~[apache-cassandra-2.1.0.jar:2.1.0]
>         at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:384)
~[apache-cassandra-2.1.0.jar:2.1.0]
>         at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:123)
~[apache-cassandra-2.1.0.jar:2.1.0]
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[apache-cassandra-2.1.0.jar:2.1.0]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_67]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_67]
>         at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
> ERROR [ValidationExecutor:39] 2014-10-13 22:20:51,148 CassandraDaemon.java:166 - Exception
in thread Thread[ValidationExecutor:39,1,main]
> java.lang.AssertionError: row DecoratedKey(-4468391015221140553, 00040000000100000800000000005abc2d00)
received out of order wrt DecoratedKey(-4436255891196926644, 0004000000010000080000000000617f4400)
>         at org.apache.cassandra.repair.Validator.add(Validator.java:136) ~[apache-cassandra-2.1.0.jar:2.1.0]
>         at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:916)
~[apache-cassandra-2.1.0.jar:2.1.0]
>         at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:100)
~[apache-cassandra-2.1.0.jar:2.1.0]
>         at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:560)
~[apache-cassandra-2.1.0.jar:2.1.0]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_67]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_67]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_67]
>         at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
> ERROR [AntiEntropySessions:34] 2014-10-13 22:20:51,149 CassandraDaemon.java:166 - Exception
in thread Thread[AntiEntropySessions:34,5,RMI Runtime]
> java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException: [repair
#1523dfa0-5327-11e4-925c-65a001aa6d80 on censored/censored, (-4468426774357114003,-4433737571269222163]]
Validation failed in /10.37.666.175
>         at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na]
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.1.0.jar:2.1.0]
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_67]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_67]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_67]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_67]
>         at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
> Caused by: org.apache.cassandra.exceptions.RepairException: [repair #1523dfa0-5327-11e4-925c-65a001aa6d80
on censored/censored, (-4468426774357114003,-4433737571269222163]] Validation failed in /10.37.666.175
>         at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:166)
~[apache-cassandra-2.1.0.jar:2.1.0]
>         at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:384)
~[apache-cassandra-2.1.0.jar:2.1.0]
>         at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:123)
~[apache-cassandra-2.1.0.jar:2.1.0]
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[apache-cassandra-2.1.0.jar:2.1.0]
>         ... 3 common frames omitted
> INFO  [MemtableFlushWriter:2295] 2014-10-13 22:20:51,180 Memtable.java:360 - Completed
flushing /ephem/data/censored/censored-f69e4e9049ac11e4bc96bfea6c7ad14a/censored-censored-ka-3185-Data.db
(126124 bytes) for commitlog position ReplayPosition(segmentId=1412194655943, position=28904247)
> INFO  [RepairJobTask:3] 2014-10-13 22:20:51,259 RepairJob.java:145 - [repair #308e2750-5327-11e4-925c-65a001aa6d80]
requesting merkle trees for censored (to [/10.167.666.137, /10.164.666.81, ip-10-37-666-175.ec2.internal/10.37.666.175])
> INFO  [AntiEntropyStage:1] 2014-10-13 22:20:51,351 RepairSession.java:171 - [repair #308e2750-5327-11e4-925c-65a001aa6d80]
Received merkle tree for censored from /10.167.666.137
> INFO  [AntiEntropyStage:1] 2014-10-13 22:20:51,446 RepairSession.java:171 - [repair #308e2750-5327-11e4-925c-65a001aa6d80]
Received merkle tree for censored from /10.164.666.81
> INFO  [ValidationExecutor:39] 2014-10-13 22:20:51,452 SSTableReader.java:383 - Opening
/ephem/data/censored/censored-f69e4e9049ac11e4bc96bfea6c7ad14a/snapshots/308e2750-5327-11e4-925c-65a001aa6d80/censored-censored-ka-3173
(87123 bytes)
> INFO  [ValidationExecutor:39] 2014-10-13 22:20:51,453 SSTableReader.java:383 - Opening
/ephem/data/censored/censored-f69e4e9049ac11e4bc96bfea6c7ad14a/snapshots/308e2750-5327-11e4-925c-65a001aa6d80/censored-censored-ka-3181
(7010 bytes)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message