Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Thu, 3 Aug 2017 22:38:00 +0000 (UTC)
From: "Jaydeepkumar Chovatia (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.13092315.1501797852000.89208.1501799880279@Atlassian.JIRA>
In-Reply-To: <JIRA.13092315.1501797852000@Atlassian.JIRA>
References: <JIRA.13092315.1501797852000@Atlassian.JIRA> <JIRA.13092315.1501797852527@jira-lw-us.apache.org>
Subject: [jira] [Updated] (CASSANDRA-13740) Orphan hint file gets created
 while node is being removed from cluster
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Thu, 03 Aug 2017 22:38:05 -0000


     [ https://issues.apache.org/jira/browse/CASSANDRA-13740?page=3Dcom.atl=
assian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jaydeepkumar Chovatia updated CASSANDRA-13740:
----------------------------------------------
    Attachment: 13740-3.0.15.txt

Please find patch with this fix attached.

> Orphan hint file gets created while node is being removed from cluster
> ----------------------------------------------------------------------
>
>                 Key: CASSANDRA-13740
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1374=
0
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Jaydeepkumar Chovatia
>             Fix For: 3.0.15
>
>         Attachments: 13740-3.0.15.txt, gossip_hang_test.py
>
>
> I have found this new issue during my test, whenever node is being remove=
d then hint file for that node gets written and stays inside the hint direc=
tory forever. I debugged the code and found that it is due to the race cond=
ition between [HintsWriteExecutor.java::flush | https://github.com/apache/c=
assandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteE=
xecutor.java#L195] and [HintsWriteExecutor.java::closeWriter | https://gith=
ub.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hi=
nts/HintsWriteExecutor.java#L106]
> .=20
> =20
> *Time t1* Node is down, as a result Hints are being written by [HintsWrit=
eExecutor.java::flush | https://github.com/apache/cassandra/blob/cassandra-=
3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
> *Time t2* Node is removed from cluster as a result it calls [HintsService=
.java-exciseStore | https://github.com/apache/cassandra/blob/cassandra-3.0/=
src/java/org/apache/cassandra/hints/HintsService.java#L327] which removes h=
int files for the node being removed
> *Time t3* Mutation stage keeps pumping Hints through [HintService.java::w=
rite | https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/=
apache/cassandra/hints/HintsService.java#L145] which again calls [HintsWrit=
eExecutor.java::flush | https://github.com/apache/cassandra/blob/cassandra-=
3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215] and n=
ew orphan file gets created
> I was writing a new dtest for {CASSANDRA-13562, CASSANDRA-13308} and that=
 helped me reproduce this new bug. I will submit patch for this new dtest l=
ater.
> I also tried following to check how this orphan hint file responds:
> 1. I tried {{nodetool truncatehints <node>}} but it fails as node is no l=
onger part of the ring
> 2. I then tried {{nodetool truncatehints}}, that still doesn=E2=80=99t re=
move hint file because it is not yet included in the [dispatchDequeue | htt=
ps://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cas=
sandra/hints/HintsStore.java#L53]
> Reproducible steps:
> Please find dTest python file {{gossip_hang_test.py}} attached which repr=
oduces this bug.
> Solution:
> This is due to race condition as mentioned above. Since {{HintsWriteExecu=
tor.java}} creates thread pool with only 1 worker, so solution becomes litt=
le simple. Whenever we [HintService.java::excise | https://github.com/apach=
e/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsSer=
vice.java#L303] a host, just store it in-memory, and check for already evic=
ted host inside [HintsWriteExecutor.java::flush | https://github.com/apache=
/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWrit=
eExecutor.java#L215]. If already evicted host is found then ignore hints.
> Jaydeep


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org