cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brandon Williams (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-5367) Hints stuck on compaction
Date Wed, 20 Mar 2013 16:37:16 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13607826#comment-13607826
] 

Brandon Williams commented on CASSANDRA-5367:
---------------------------------------------

It looks like hints aren't stuck, there's a thread trying to deliver to a host and there's
a large compaction of hints going on.  The host that the hints are for is the problem.
                
> Hints stuck on compaction
> -------------------------
>
>                 Key: CASSANDRA-5367
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5367
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 1.2.2
>         Environment: 80 Node cluster on 1.2.2 (problem has been around since before 1.0)
>            Reporter: Brooke Bryan
>         Attachments: thread.log
>
>
> When our cluster is handling hints, we will very often see hints get stuck on nodes if
it is unable to communicate with another node.  The problem is not that the other node is
down, the other node will be sat doing compactions, or running out of memory.  While that
node is a problem, and needs to be fixed, all other nodes on the cluster will stick waiting
to handle hints between that node and itself.
> This causes a pretty major knock on effect throughout the entire cluster, causing hints
to back up.  We are seeing some nodes backed up with 14GB of hints, after 2 days of the hints
being stuck.
> Also, during this "stuck" session, compactionstats will show a compaction on the system
hints column family, and not change the completed bytes amount.
> This is the only reason for an entire cluster to get very bogged down from what I have
experienced, and requires a lot of manual intervention to get everything back online.
> After putting a node into debug mode, I have narrowed down the issue to be within:
> startColumn = hint.name(); (line ~361 HintedHandoffManager) and line 390
> based on the log output, and through pausing handoffs etc.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message