cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fredrik L Stigbäck (Created) (JIRA) <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-3546) Hinted handoffs isn't delivered if/when HintedHandOffManager ends up in invalid state.
Date Thu, 01 Dec 2011 10:43:40 GMT
Hinted handoffs isn't delivered if/when HintedHandOffManager ends up in invalid state.
--------------------------------------------------------------------------------------

                 Key: CASSANDRA-3546
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3546
             Project: Cassandra
          Issue Type: Bug
          Components: Core
    Affects Versions: 1.0.3
            Reporter: Fredrik L Stigbäck


Running Cassandra 1.0.3.
I've done some testing with 2 nodes (node A, node B), replication factor 2.
I take node A down, writing some data to node B and then take node A up.
Sometimes hints aren't delivered when node A comes up.

I've done some debugging in org.apache.cassandra.db.HintedHandOffManager and sometimes node
B ends up in a strange state in method org.apache.cassandra.db.HintedHandOffManager.deliverHints(final
InetAddress to), where org.apache.cassandra.db.HintedHandOffManager.queuedDeliveries already
has node A in it's Set and therefore no hints will ever be delivered to node A.

The only reason for this that I can see is that in org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(InetAddress
endpoint) the hintStore.isEmpty() check returns true and the endpoint (node A)  isn't removed
from org.apache.cassandra.db.HintedHandOffManager.queuedDeliveries. Then no hints will ever
be delivered again until node B is restarted.

During what conditions will hintStore.isEmpty() return true?
Shouldn't the hintStore.isEmpty() check be inside the try {} finally{} clause, removing the
endpoint from queuedDeliveries in the finally block?

{code}
public void deliverHints(final InetAddress to)
{
    logger_.debug("deliverHints to {}", to);
    if (!queuedDeliveries.add(to))
        return;
    .......
}
{code}

{code}
private void deliverHintsToEndpoint(InetAddress endpoint) 
    throws IOException, DigestMismatchException, InvalidRequestException, TimeoutException,
InterruptedException
{
     ColumnFamilyStore hintStore = Table.open(Table.SYSTEM_TABLE).getColumnFamilyStore(HINTS_CF);
     if (hintStore.isEmpty())
         return; // nothing to do, don't confuse users by logging a no-op handoff
     try
     {
         ......
     }
     finally
     {
         queuedDeliveries.remove(endpoint);
     }
}
{code} 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message