cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcus Eriksson (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-3624) Hinted Handoff - related OOM
Date Fri, 23 Dec 2011 06:11:30 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-3624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175302#comment-13175302
] 

Marcus Eriksson commented on CASSANDRA-3624:
--------------------------------------------

not yet, i might be able to do it today though

this happened in a production cluster so i will have to try to reproduce somewhere else
                
> Hinted Handoff - related OOM
> ----------------------------
>
>                 Key: CASSANDRA-3624
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3624
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 1.0.0
>            Reporter: Marcus Eriksson
>            Assignee: Jonathan Ellis
>              Labels: hintedhandoff
>             Fix For: 1.0.7
>
>         Attachments: 3624.txt
>
>
> One of our nodes had collected alot of hints for another node, so when the dead node
came back and the row mutations were read back from disk, the node died with an OOM-exception
(and kept dying after restart, even with increased heap (from 8G to 12G)). The heap dump contained
alot of SuperColumns and our application does not use those (but HH does). 
> I'm guessing that each mutation is big so that PAGE_SIZE*<mutation_size> does not
fit in memory (will check this tomorrow)
> A simple fix (if my assumption above is correct) would be to reduce the PAGE_SIZE in
HintedHandOffManager.java to something like 10 (or even 1?) to reduce the memory pressure.
The performance hit would be small since we are doing the hinted handoff throttle delay sleep
before sending every *mutation* anyway (not every page), thoughts?
> If anyone runs in to the same problem, I got the node started again by simply removing
the HintsColumnFamily* files.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message