incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brandon Williams <dri...@gmail.com>
Subject Re: Ghost node showing up in the ring
Date Wed, 23 Mar 2011 23:04:23 GMT
On Wed, Mar 23, 2011 at 2:23 PM, Alexis Lê-Quôc <alq@datadoghq.com> wrote:

>  I'm going to take a stab at a hypothesis:
> Sunday: I drain and decommission 2.3.4.193 *but* I forget to run node
> cleanup on the rest of the nodes. The ring looks clean but I did not see
> "Annoucing that ..." in the logs.
>
> Tuesday: the ghost node reappears on the ring for all nodes.
>
> Could this be caused by old hinted handoffs for 2.3.4.193 that were
> processed at that time, causing the rest of the nodes to think that the
> 2.3.4.193 is still present (albeit down)?
>

No.  See my hypothesis and patch on
https://issues.apache.org/jira/browse/CASSANDRA-2371  The Sun-Tues
connection here is revealing, because gossiper's aVeryLongTime is 3 days.

Should cleanup be run periodically? I run repair every few days (my
> gcgraceperiod is 10 days).
>

You only need to run cleanup after topology changes.  Hints expire
themselves at gcgrace.

-Brandon

Mime
View raw message