cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paulo Motta (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-8523) Writes should be sent to a replacement node while it is streaming in data
Date Fri, 29 Jul 2016 20:18:20 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-8523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399941#comment-15399941
] 

Paulo Motta commented on CASSANDRA-8523:
----------------------------------------

Thanks for taking a look! Created CASSANDRA-12344 to follow-up with support for this when
the replacement node has the same address as the original node.

Rebased patch and dtests as well as merged up to 3.0+. All patches and CI results available
below:

||2.2||3.0||3.9||trunk||dtest||
|[branch|https://github.com/apache/cassandra/compare/cassandra-2.2...pauloricardomg:2.2-8523]|[branch|https://github.com/apache/cassandra/compare/cassandra-3.0...pauloricardomg:3.0-8523]|[branch|https://github.com/apache/cassandra/compare/cassandra-3.9...pauloricardomg:3.9-8523]|[branch|https://github.com/apache/cassandra/compare/trunk...pauloricardomg:trunk-8523]|[branch|https://github.com/riptano/cassandra-dtest/compare/master...pauloricardomg:8523]|
|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.2-8523-testall/lastCompletedBuild/testReport/]|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-3.0-8523-testall/lastCompletedBuild/testReport/]|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-3.9-8523-testall/lastCompletedBuild/testReport/]|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-trunk-8523-testall/lastCompletedBuild/testReport/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.2-8523-dtest/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-3.0-8523-dtest/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-3.9-8523-dtest/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-trunk-8523-dtest/lastCompletedBuild/testReport/]|

There were some minor merge conflicts on 3.0, and a slightly larger conflict on 3.9 due to
CASSANDRA-10134, so I did some refactoring in the 3.9+ version to move most of the logic to
{{prepareForReplacement}}. Can you take another look [~rlow]?

Dtest PR created [here|https://github.com/riptano/cassandra-dtest/pull/1155].

While this is marked an improvement and would theoretically only go to trunk, this limitation
is pretty counter-intuitive and probably hurts many users in the wild, and since the changeset
is relatively small and self-contained, I think it could be interpreted as a bugfix and perhaps
go on 2.2+ or maybe 3.0+. WDYT [~brandon.williams] [~jkni] ?

> Writes should be sent to a replacement node while it is streaming in data
> -------------------------------------------------------------------------
>
>                 Key: CASSANDRA-8523
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8523
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Richard Wagner
>            Assignee: Paulo Motta
>             Fix For: 2.1.x
>
>
> In our operations, we make heavy use of replace_address (or replace_address_first_boot)
in order to replace broken nodes. We now realize that writes are not sent to the replacement
nodes while they are in hibernate state and streaming in data. This runs counter to what our
expectations were, especially since we know that writes ARE sent to nodes when they are bootstrapped
into the ring.
> It seems like cassandra should arrange to send writes to a node that is in the process
of replacing another node, just like it does for a nodes that are bootstraping. I hesitate
to phrase this as "we should send writes to a node in hibernate" because the concept of hibernate
may be useful in other contexts, as per CASSANDRA-8336. Maybe a new state is needed here?
> Among other things, the fact that we don't get writes during this period makes subsequent
repairs more expensive, proportional to the number of writes that we miss (and depending on
the amount of data that needs to be streamed during replacement and the time it may take to
rebuild secondary indexes, we could miss many many hours worth of writes). It also leaves
us more exposed to consistency violations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message