cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sylvain Lebresne (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-2433) Failed Streams Break Repair
Date Thu, 09 Jun 2011 14:09:59 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sylvain Lebresne updated CASSANDRA-2433:
----------------------------------------

    Attachment: 0004-Reports-validation-compaction-errors-back-to-repair-v3.patch
                0003-Report-streaming-errors-back-to-repair-v3.patch
                0002-Register-in-gossip-to-handle-node-failures-v3.patch
                0001-Put-repair-session-on-a-Stage-and-add-a-method-to-re-v3.patch

Attaching v3 rebased (on 0.8).

bq. Since we're not trying to control throughput or monitor sessions, could we just use Stage.MISC?

The thing is that repair session are very long lived. And MISC is single threaded. So that
would block other task that are not supposed to block. We could make MISC multi-threaded but
even then it's not a good idea to mix short lived and long lived task on the same stage.

bq. I think RepairSession.exception needs to be volatile to ensure that the awoken thread
sees it

Done in v3.

bq. Would it be better if RepairSession implemented IEndpointStateChangeSubscriber directly?

Good idea, it's slightly simpler, done in v3.

bq. The endpoint set needs to be threadsafe, since it will be modified by the endpoint state
change thread, and the AE_STAGE thread

Done in v3. That will probably change with CASSANDRA-2610 anyway (which I have to update)

bq. Should StreamInSession.retries be volatile/atomic? (likely they won't retry quickly enough
for it to be a problem, but...)

I did not change that, but if it's a problem for retries to not be volatile, I suspect having
StreamInSession.current not volatile is also a problem. But really I'd be curious to see that
be a problem.

bq. Playing devil's advocate: would sending a half-built tree in case of failure still be
useful?

I don't think it is. Or more precisely, if you do send half-built tree, you'll have to be
careful that the other doesn't consider what's missing as ranges not being in sync (I don't
think people will be happy with tons of data being stream just because we happen to have a
bug that make compaction throw an exception during the validation). So I think you cannot
do much with a half-built tree, and it will add complication. For a case where people will
need to restart a repair anyway once whatever happened is fixed

bq. success might need to be volatile as well

Done in v3.


> Failed Streams Break Repair
> ---------------------------
>
>                 Key: CASSANDRA-2433
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2433
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Benjamin Coverston
>            Assignee: Sylvain Lebresne
>              Labels: repair
>             Fix For: 0.8.1
>
>         Attachments: 0001-Put-repair-session-on-a-Stage-and-add-a-method-to-re-v2.patch,
0001-Put-repair-session-on-a-Stage-and-add-a-method-to-re-v3.patch, 0001-Put-repair-session-on-a-Stage-and-add-a-method-to-re.patch,
0002-Register-in-gossip-to-handle-node-failures-v2.patch, 0002-Register-in-gossip-to-handle-node-failures-v3.patch,
0002-Register-in-gossip-to-handle-node-failures.patch, 0003-Report-streaming-errors-back-to-repair-v2.patch,
0003-Report-streaming-errors-back-to-repair-v3.patch, 0003-Report-streaming-errors-back-to-repair.patch,
0004-Reports-validation-compaction-errors-back-to-repair-v2.patch, 0004-Reports-validation-compaction-errors-back-to-repair-v3.patch,
0004-Reports-validation-compaction-errors-back-to-repair.patch
>
>
> Running repair in cases where a stream fails we are seeing multiple problems.
> 1. Although retry is initiated and completes, the old stream doesn't seem to clean itself
up and repair hangs.
> 2. The temp files are left behind and multiple failures can end up filling up the data
partition.
> These issues together are making repair very difficult for nearly everyone running repair
on a non-trivial sized data set.
> This issue is also being worked on w.r.t CASSANDRA-2088, however that was moved to 0.8
for a few reasons. This ticket is to fix the immediate issues that we are seeing in 0.7.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message