cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paolo Crosato <paolo.cros...@targaubiest.com>
Subject Re: nodetool repair stalled
Date Mon, 13 Jan 2014 09:25:01 GMT
Hi,

I rebooted the nodes and started a fresh repair session. The repair 
session was started on node 1.

This time actually I got this error on the node that started the repair:

ERROR [AntiEntropySessions:2] 2014-01-10 09:44:46,360 RepairSession.java 
(line 278) [repair #728f4860-79d3-11e3-8c98-a1357a0d9222] session 
completed with the following error
org.apache.cassandra.exceptions.RepairException: [repair 
#728f4860-79d3-11e3-8c98-a1357a0d9222 on OpsCenter/rollups300, 
(4515884230644880127,4556138740897423021]] Sync failed between 
/10.255.235.18 and /10.255.235.19
     at 
org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:200)
     at 
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:193)
     at 
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:59)
     at 
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:60)
     at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
     at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
     at java.lang.Thread.run(Thread.java:744)
ERROR [AntiEntropySessions:2] 2014-01-10 09:44:46,399 
CassandraDaemon.java (line 187) Exception in thread 
Thread[AntiEntropySessions:2,5,RMI Runtime]
java.lang.RuntimeException: 
org.apache.cassandra.exceptions.RepairException: [repair 
#728f4860-79d3-11e3-8c98-a1357a0d9222 on OpsCenter/rollups300, 
(4515884230644880127,4556138740897423021]] Sync failed between 
/10.255.235.18 and /10.255.235.19
     at com.google.common.base.Throwables.propagate(Throwables.java:160)
     at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
     at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
     at java.util.concurrent.FutureTask.run(FutureTask.java:262)
     at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
     at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
     at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.cassandra.exceptions.RepairException: [repair 
#728f4860-79d3-11e3-8c98-a1357a0d9222 on OpsCenter/rollups300, 
(4515884230644880127,4556138740897423021]] Sync failed between 
/10.255.235.18 and /10.255.235.19
     at 
org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:200)
     at 
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:193)
     at 
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:59)
     at 
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:60)
     ... 3 more

On the other node i left some black lines between these timestamps:

INFO [ValidationExecutor:3] 2014-01-10 09:42:41,320 SSTableReader.java 
(line 223) Opening 
/data/cassandra/data/OpsCenter/rollups60/snapshots/29e4d5d0-79d3-11e3-8c98-a1357a0d9222/OpsCenter-rollups60-jb-11522

(88 bytes)

  INFO [ValidationExecutor:14] 2014-01-10 10:37:48,509 
SSTableReader.java (line 223) Opening 
/data/cassandra/data/OpsCenter/rollups60/snapshots/d5176b00-79da-11e3-8c98-a1357a0d9222/OpsCenter-rollups60-jb-16275

(493003 b

Between I have many log files full of "Opening ...." logs.

I've noticed the repair sessions seems always to hang on the opscenter 
keyspace. Would uninstall/reinstall help resolve the issue?

Anyway, I attached the logs for the nodes involved, I'm sorry if there 
is a lot of noise.

Thanks for any input.

Regards,

Paolo Crosato

Il 09/01/2014 03:54, sankalp kohli ha scritto:
> Hi,
>     Can you attach the logs around repair. Please do that for node 
> which triggered it and nodes involved in repair. I will try to find 
> something useful.
>
> Thanks,
> Sankalp
>
>
> On Wed, Jan 8, 2014 at 10:18 AM, Robert Coli <rcoli@eventbrite.com 
> <mailto:rcoli@eventbrite.com>> wrote:
>
>     On Wed, Jan 8, 2014 at 8:52 AM, Paolo Crosato
>     <paolo.crosato@targaubiest.com
>     <mailto:paolo.crosato@targaubiest.com>> wrote:
>
>         I have two nodes with Cassandra 2.0.3, where repair sessions
>         hang for an undefinite time. I'm running nodetool repair once
>         a week on every node, on different days. Currently I have like
>         4 repair sessions running on each node, one since 3 weeks and
>         none has finished.
>         Reading the logs I didn't find any exception, apparently one
>         of the repair session got stuck at this command:
>
>         Has anybody any suggestion on why a nodetool repair might be
>         stuck and how to debug it?
>
>
>     Cassandra repair has never quite worked right. It got a wholesale
>     re-write in 2.0.x and "should" be more robust and at very least
>     log more than before. But unfortunately I have heard a few reports
>     like yours, so it is probably not completely fixed.
>
>     That said, that only option you have for failed repairs seems to
>     be to restart the affected nodes. Your input as an operator of
>     2.0.x who would appreciate an alternative is welcome at :
>
>     https://issues.apache.org/jira/browse/CASSANDRA-3486
>
>     =Rob
>
>


-- 
Paolo Crosato
Software engineer/Custom Solutions
e-mail: paolo.crosato@targaubiest.com
Office phone: +3904221722825

UBIEST S.p.A.
........................................................................................
www.ubiest.com
Via E. Reginato, 85/H - 31100 Treviso- ITALY Tel [+39] 0422 210 194 - Fax [+39] 0422 210 270
........................................................................................
This message is for the designated recipient only and may contain privileged, proprietary,
or otherwise private information. If you have received it in error, please notify the sender
immediately and delete the original. Any other use of the email by you is prohibited.


Mime
View raw message