cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rafael Loureiro <rafael.loure...@emailage.com>
Subject Repair service fails with sync failed between X and Y
Date Fri, 20 Nov 2015 00:45:14 GMT
Hi All,


First, let me apologize for the long message. Repair service is driving me
insane. :)


We are running Cassandra version *2.1.11.908* in 3 data centers. While
running nodetool repair we are receiving the following error:


[2015-11-20 00:25:24,803] Repair session
2ed2f930-8f1d-11e5-a498-4b9679ec178d for range
(4375311677306593380,4384065392379704564] failed with error
org.apache.cassandra.exceptions.RepairException:

[repair #2ed2f930-8f1d-11e5-a498-4b9679ec178d on keyspace/table,
(4375311677306593380,4384065392379704564]] Sync failed between /10.10.8.104
and /10.30.30.6


Unfortunately, not much information online. Any ideas how to fix this issue
or determine root cause?


Thanks!


Detailed log below:


ERROR [AntiEntropySessions:8] 2015-11-20 00:25:24,801
RepairSession.java:303 - [repair #2ed2f930-8f1d-11e5-a498-4b9679ec178d]
session completed with the following
error org.apache.cassandra.exceptions.RepairException: [repair
#2ed2f930-8f1d-11e5-a498-4b9679ec178d on keyspace/table,
(4375311677306593380,4384065392379704564]] Sync failed between /10.10.8.104
and /10.30.30.6 at
org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:223)
~[cassandra-all-2.1.11.908.jar:2.1.11.908] at
 org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:411)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]

        at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:134)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]

        at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source) [na:1.8.0_60]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source) [na:1.8.0_60]

        at java.lang.Thread.run(Unknown Source) [na:1.8.0_60]

ERROR [AntiEntropySessions:8] 2015-11-20 00:25:24,802
CassandraDaemon.java:227 - Exception in thread
Thread[AntiEntropySessions:8,5,RMI Runtime]

java.lang.RuntimeException:
org.apache.cassandra.exceptions.RepairException: [repair
#2ed2f930-8f1d-11e5-a498-4b9679ec178d on keyspace/table,
(4375311677306593380,4384065392379704564]] Sync failed between /10.10.8.104
and /10.30.30.6

        at com.google.common.base.Throwables.propagate(Throwables.java:160)
~[guava-16.0.1.jar:na]

        at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]

        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source) ~[na:1.8.0_60]

        at java.util.concurrent.FutureTask.run(Unknown Source)
~[na:1.8.0_60]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source) [na:1.8.0_60]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source) [na:1.8.0_60]

        at java.lang.Thread.run(Unknown Source) [na:1.8.0_60]

Caused by: org.apache.cassandra.exceptions.RepairException: [repair
#2ed2f930-8f1d-11e5-a498-4b9679ec178d on keyspace/table,
(4375311677306593380,4384065392379704564]] Sync failed between /10.10.8.104
and /10.30.30.6

        at
org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:223)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]

        at
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:411)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]

        at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:134)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]

        at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]

        ... 3 common frames omitted

ERROR [Thread-3128] 2015-11-20 00:25:24,802  StorageService.java:3008 -
Repair session 2ed2f930-8f1d-11e5-a498-4b9679ec178d for range
(4375311677306593380,4384065392379704564] failed with error
org.apache.cassandra.exceptions.RepairException: [repair
#2ed2f930-8f1d-11e5-a498-4b9679ec178d on keyspace/table,
(4375311677306593380,4384065392379704564]] Sync failed between /10.10.8.104
and /10.30.30.6

java.util.concurrent.ExecutionException: java.lang.RuntimeException:
org.apache.cassandra.exceptions.RepairException: [repair
#2ed2f930-8f1d-11e5-a498-4b9679ec178d on keyspace/table,
(4375311677306593380,4384065392379704564]] Sync failed between /10.10.8.104
and /10.30.30.6

        at java.util.concurrent.FutureTask.report(Unknown Source)
[na:1.8.0_60]

        at java.util.concurrent.FutureTask.get(Unknown Source) [na:1.8.0_60]

        at
org.apache.cassandra.service.StorageService$4.runMayThrow(StorageService.java:2999)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]

        at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
[cassandra-all-2.1.11.908.jar:2.1.11.908]

        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source) [na:1.8.0_60]

        at java.util.concurrent.FutureTask.run(Unknown Source) [na:1.8.0_60]

        at java.lang.Thread.run(Unknown Source) [na:1.8.0_60]

Caused by: java.lang.RuntimeException:
org.apache.cassandra.exceptions.RepairException: [repair
#2ed2f930-8f1d-11e5-a498-4b9679ec178d on keyspace/table,
(4375311677306593380,4384065392379704564]] Sync failed between /10.10.8.104
and /10.30.30.6

        at com.google.common.base.Throwables.propagate(Throwables.java:160)
~[guava-16.0.1.jar:na]

        at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
[cassandra-all-2.1.11.908.jar:2.1.11.908]

        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source) [na:1.8.0_60]

        at java.util.concurrent.FutureTask.run(Unknown Source) [na:1.8.0_60]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source) ~[na:1.8.0_60]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source) ~[na:1.8.0_60]

        ... 1 common frames omitted

Caused by: org.apache.cassandra.exceptions.RepairException: [repair
#2ed2f930-8f1d-11e5-a498-4b9679ec178d on keyspace/table,
(4375311677306593380,4384065392379704564]] Sync failed between /10.10.8.104
and /10.30.30.6

        at
org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:223)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]

        at
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:411)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]

        at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:134)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]

        at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]


        ... 3 common frames omitted

Thanks,


Rafael

Mime
View raw message