cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Terrance Shepherd (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-5083) node locks up when streamed hinted handoffs and sending node dies.
Date Thu, 20 Dec 2012 18:31:14 GMT
Terrance Shepherd created CASSANDRA-5083:
--------------------------------------------

             Summary: node locks up when streamed hinted handoffs and sending node dies.
                 Key: CASSANDRA-5083
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5083
             Project: Cassandra
          Issue Type: Bug
            Reporter: Terrance Shepherd
         Attachments: failure.log

Last night 2 nodes appeared to have died do to an issues while one node was streaming hinted
hand offs to another node. Attached is the last 100 lines from the node that was receiving
hints. The version of Cassandra is 1.1.6.

The order of events appeared to be
1) a node was unavailable ( unknown reason ) for a period of time causing hinted hand offs
to back up on another node.
2) the node that went down came back on line and hinted hand offs started transferring.
3) the node holding the hints's disk space grew exponentially and filled up the data directory
holding the hinted hand off column family. the error long is full of unable to compact hinted
column family keyspaces.
4) the node holding the hints became unavailable.
5) the node being streamed the hints locked up. cassandra was still "running" but not responding
to requests, gossip or restarts

Terrance
Developer
Dynamic Network Services, Inc.
http://www.dyn.com


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message