cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eldon Stegall (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-4162) nodetool disablegossip does not prevent gossip delivery of writes via already-initiated hinted handoff
Date Tue, 17 Apr 2012 22:14:18 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255993#comment-13255993
] 

Eldon Stegall commented on CASSANDRA-4162:
------------------------------------------

I am a bit fuzzy on the internals of when a HH session starts and stops. However, I have seen
similar behavior, and specifically in situations where a very intensive, very long-running
compaction is occuring, some sort of thrashing appears to happen, and neither the HH nor the
compaction finish. In a situation (perhaps an edge case) where you want to isolate a node
in order to let a very-long-running compaction to complete, you may not want to kill and restart
the node, as that could dramatically increase your time to rejoin the ring (particularly if
you have already finished a significant portion of the compaction). I just shut it all off
with iptables like so:
sudo iptables -A INPUT -p tcp --dport 7000 -j DROP
sudo iptables -A INPUT -p tcp --dport 9160 -j DROP
sudo iptables -A OUTPUT -p tcp --dport 9160 -j DROP
sudo iptables -A OUTPUT -p tcp --dport 7000 -j DROP

It's not pretty, but it works, and I think maybe it all goes away with leveldb, if only I
had the cycles to switch us to that. Forgive me if this seems odd, I have had my head out
of cassandra for a little while now. My 2 cents.
                
> nodetool disablegossip does not prevent gossip delivery of writes via already-initiated
hinted handoff
> ------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-4162
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4162
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.9
>         Environment: reported on IRC, believe it was a linux environment, nick "rhone",
cassandra 1.0.8
>            Reporter: Robert Coli
>            Priority: Minor
>              Labels: gossip
>
> This ticket derives from #cassandra, aaron_morton and I assisted a user who had run "disablethrift"
and "disablegossip" and was confused as to why he was seeing writes to his node.
> Aaron and I went through a series of debugging questions, user verified that there was
traffic on the gossip port. His node was showing as down from the perspective of other nodes,
and nodetool also showed that gossip was not active.
> Aaron read the code and had the user turn debug logging on. The user saw Hinted Handoff
messages being delivered and Aaron confirmed in the code that a hinted handoff delivery session
only checks gossip state when it first starts. As a result, it will continue to deliver hints
and disregard gossip state on the target node.
> per nodetool docs
> "
> disablegossip          - Disable gossip (effectively marking the node dead)
> "
> I believe most people will be using disablegossip and disablethrift for operational reasons,
and propose that they do not expect HH delivery to continue, via gossip, when they have run
"disablegossip".

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message