Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Fri, 15 Jan 2016 18:48:39 +0000 (UTC)
From: "sankalp kohli (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12922589.1450326454000.124808.1452883719928@Atlassian.JIRA>
In-Reply-To: <JIRA.12922589.1450326454000@Atlassian.JIRA>
References: <JIRA.12922589.1450326454000@Atlassian.JIRA>
 <JIRA.12922589.1450326454151@arcas>
Subject: [jira] [Updated] (CASSANDRA-10887) Pending range calculator gives
 wrong pending ranges for moves
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


     [ https://issues.apache.org/jira/browse/CASSANDRA-10887?page=3Dcom.atl=
assian.jira.plugin.system.issuetabpanels:all-tabpanel ]

sankalp kohli updated CASSANDRA-10887:
--------------------------------------
    Since Version: 0.8 beta 1  (was: 2.0.16)

> Pending range calculator gives wrong pending ranges for moves
> -------------------------------------------------------------
>
>                 Key: CASSANDRA-10887
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1088=
7
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Coordination
>            Reporter: Richard Low
>            Assignee: sankalp kohli
>            Priority: Critical
>             Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
>         Attachments: CASSANDRA-10887.diff, CASSANDRA_10887_2.2.diff, CASS=
ANDRA_10887_3.0.diff, CASSANDRA_10887_v2.diff, CASSANDRA_10887_v3.diff
>
>
> My understanding is the PendingRangeCalculator is meant to calculate who =
should receive extra writes during range movements. However, it adds the wr=
ong ranges for moves. An extreme example of this can be seen in the followi=
ng reproduction. Create a 5 node cluster (I did this on 2.0.16 and 2.2.4) a=
nd a keyspace RF=3D3 and a simple table. Then start moving a node and immed=
iately kill -9 it. Now you see a node as down and moving in the ring. Try a=
 quorum write for a partition that is stored on that node - it will fail wi=
th a timeout. Further, all CAS reads or writes fail immediately with unavai=
lable exception because they attempt to include the moving node twice. This=
 is likely to be the cause of CASSANDRA-10423.
> In my example I had this ring:
> 127.0.0.1  rack1       Up     Normal  170.97 KB       20.00%             =
 -9223372036854775808
> 127.0.0.2  rack1       Up     Normal  124.06 KB       20.00%             =
 -5534023222112865485
> 127.0.0.3  rack1       Down   Moving  108.7 KB        40.00%             =
 1844674407370955160
> 127.0.0.4  rack1       Up     Normal  142.58 KB       0.00%              =
 1844674407370955161
> 127.0.0.5  rack1       Up     Normal  118.64 KB       20.00%             =
 5534023222112865484
> Node 3 was moving to -1844674407370955160. I added logging to print the p=
ending and natural endpoints. For ranges owned by node 3, node 3 appeared i=
n pending and natural endpoints. The blockFor is increased to 3 so we=E2=80=
=99re effectively doing CL.ALL operations. This manifests as write timeouts=
 and CAS unavailables when the node is down.
> The correct pending range for this scenario is node 1 is gaining the rang=
e (-1844674407370955160, 1844674407370955160). So node 1 should be added as=
 a destination for writes and CAS for this range, not node 3.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)