Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7AC981826F for ; Sun, 28 Jun 2015 15:19:05 +0000 (UTC) Received: (qmail 49909 invoked by uid 500); 28 Jun 2015 15:19:05 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 49872 invoked by uid 500); 28 Jun 2015 15:19:05 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 49859 invoked by uid 99); 28 Jun 2015 15:19:05 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 28 Jun 2015 15:19:05 +0000 Date: Sun, 28 Jun 2015 15:19:05 +0000 (UTC) From: "david (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Comment Edited] (CASSANDRA-9668) RepairException when trying to run concurrent repair -pr MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14604718#comment-14604718 ] david edited comment on CASSANDRA-9668 at 6/28/15 3:19 PM: ----------------------------------------------------------- Yes, here is corrosponding error: {noformat} ERROR [ValidationExecutor:19] 2015-06-28 09:33:12,261 CompactionManager.java:972 - Cannot start multiple repair sessions over the same sstables ERROR [ValidationExecutor:19] 2015-06-28 09:33:12,261 Validator.java:245 - Failed creating a merkle tree for [repair #b1e67660-1d78-11e5-aec7-4f05493cbe02 on evosload_services_otg_scee_com_driveclub/data, (-4660677346721084182,-4658765298409301171]], /172.31.46.189 (see log for details) ERROR [ValidationExecutor:19] 2015-06-28 09:33:12,261 CassandraDaemon.java:223 - Exception in thread Thread[ValidationExecutor:19,1,main] java.lang.RuntimeException: Cannot start multiple repair sessions over the same sstables at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:973) ~[apache-cassandra-2.1.7.jar:2.1.7] at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:94) ~[apache-cassandra-2.1.7.jar:2.1.7] at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:623) ~[apache-cassandra-2.1.7.jar:2.1.7] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_40] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_40] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_40] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40] {noformat} This suggest running concurrent repairs (with -pr) is not possible. Is this true? was (Author: biffta): Yes, here is corrosponding error: {noformat} ERROR [ValidationExecutor:19] 2015-06-28 09:33:12,029 CompactionManager.java:972 - Cannot start multiple repair sessions over the same sstables ERROR [ValidationExecutor:19] 2015-06-28 09:33:12,029 Validator.java:245 - Failed creating a merkle tree for [repair #b1c30fe0-1d78-11e5-aec7-4f05493cbe02 on keyspace/data, (9062648853864216757,9072201154757474095]], /172.31.46.189 (see log for details) ERROR [ValidationExecutor:19] 2015-06-28 09:33:12,029 CassandraDaemon.java:223 - Exception in thread Thread[ValidationExecutor:19,1,main] java.lang.RuntimeException: Cannot start multiple repair sessions over the same sstables at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:973) ~[apache-cassandra-2.1.7.jar:2.1.7] at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:94) ~[apache-cassandra-2.1.7.jar:2.1.7] at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:623) ~[apache-cassandra-2.1.7.jar:2.1.7] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_40] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_40] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_40] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40] {noformat} This suggest running concurrent repairs (with -pr) is not possible. Is this true? > RepairException when trying to run concurrent repair -pr > -------------------------------------------------------- > > Key: CASSANDRA-9668 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9668 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Cassandra 2.1.7 > Reporter: david > Assignee: Yuki Morishita > Priority: Critical > Fix For: 2.1.x > > > Was on 2.1.3 having very similar issues to those described in: > https://issues.apache.org/jira/browse/CASSANDRA-9266 > I updated to 2.1.7, more for some other fixes, but now if I try and run concurrent repairs (different boxes) consistently get: > {noformat} > ERROR [Thread-14156] 2015-06-28 09:33:12,616 StorageService.java:2959 - Repair session b1e67660-1d78-11e5-aec7-4f05493cbe02 for range (-4660677346721084182,-4658765298409301171] failed with error org.apache.cassandra.exceptions.RepairException: [repair #b1e67660-1d78-11e5-aec7-4f05493cbe02 on keyspace/data, (-4660677346721084182,-4658765298409301171]] Validation failed in /172.31.13.127 > java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException: [repair #b1e67660-1d78-11e5-aec7-4f05493cbe02 on keyspace/data, (-4660677346721084182,-4658765298409301171]] Validation failed in /172.31.13.127 > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [na:1.8.0_40] > at java.util.concurrent.FutureTask.get(FutureTask.java:192) [na:1.8.0_40] > at org.apache.cassandra.service.StorageService$4.runMayThrow(StorageService.java:2950) ~[apache-cassandra-2.1.7.jar:2.1.7] > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [apache-cassandra-2.1.7.jar:2.1.7] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_40] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_40] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40] > Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException: [repair #b1e67660-1d78-11e5-aec7-4f05493cbe02 on keyspace/data, (-4660677346721084182,-4658765298409301171]] Validation failed in /172.31.13.127 > at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na] > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) [apache-cassandra-2.1.7.jar:2.1.7] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_40] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_40] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_40] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_40] > ... 1 common frames omitted > Caused by: org.apache.cassandra.exceptions.RepairException: [repair #b1e67660-1d78-11e5-aec7-4f05493cbe02 on keyspace/data, (-4660677346721084182,-4658765298409301171]] Validation failed in /172.31.13.127 > at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:166) ~[apache-cassandra-2.1.7.jar:2.1.7] > at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:406) ~[apache-cassandra-2.1.7.jar:2.1.7] > at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:134) ~[apache-cassandra-2.1.7.jar:2.1.7] > at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.7.jar:2.1.7] > ... 3 common frames omitted > {noformat} > The specific repair command being issued: > {noformat} > nodetool repair -local -pr -inc -par -- keyspace & > {noformat} > It's a 15 box environment with a replication factor of 3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)