cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fredrik L Stigbäck (Created) (JIRA) <>
Subject [jira] [Created] (CASSANDRA-3546) Hinted handoffs isn't delivered if/when HintedHandOffManager ends up in invalid state.
Date Thu, 01 Dec 2011 10:43:40 GMT
Hinted handoffs isn't delivered if/when HintedHandOffManager ends up in invalid state.

                 Key: CASSANDRA-3546
             Project: Cassandra
          Issue Type: Bug
          Components: Core
    Affects Versions: 1.0.3
            Reporter: Fredrik L Stigbäck

Running Cassandra 1.0.3.
I've done some testing with 2 nodes (node A, node B), replication factor 2.
I take node A down, writing some data to node B and then take node A up.
Sometimes hints aren't delivered when node A comes up.

I've done some debugging in org.apache.cassandra.db.HintedHandOffManager and sometimes node
B ends up in a strange state in method org.apache.cassandra.db.HintedHandOffManager.deliverHints(final
InetAddress to), where org.apache.cassandra.db.HintedHandOffManager.queuedDeliveries already
has node A in it's Set and therefore no hints will ever be delivered to node A.

The only reason for this that I can see is that in org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(InetAddress
endpoint) the hintStore.isEmpty() check returns true and the endpoint (node A)  isn't removed
from org.apache.cassandra.db.HintedHandOffManager.queuedDeliveries. Then no hints will ever
be delivered again until node B is restarted.

During what conditions will hintStore.isEmpty() return true?
Shouldn't the hintStore.isEmpty() check be inside the try {} finally{} clause, removing the
endpoint from queuedDeliveries in the finally block?

public void deliverHints(final InetAddress to)
    logger_.debug("deliverHints to {}", to);
    if (!queuedDeliveries.add(to))

private void deliverHintsToEndpoint(InetAddress endpoint) 
    throws IOException, DigestMismatchException, InvalidRequestException, TimeoutException,
     ColumnFamilyStore hintStore =;
     if (hintStore.isEmpty())
         return; // nothing to do, don't confuse users by logging a no-op handoff

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message