couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Kocoloski (JIRA)" <j...@apache.org>
Subject [jira] Commented: (COUCHDB-888) out of memory crash when compacting document with lots of edit branches
Date Thu, 16 Sep 2010 21:35:32 GMT

    [ https://issues.apache.org/jira/browse/COUCHDB-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12910328#action_12910328
] 

Adam Kocoloski commented on COUCHDB-888:
----------------------------------------

Hi Damien, I guess the implications of limiting the number of edit branches would be pretty
serious, right?  When we stem a branch we run the risk of introducing spurious conflicts,
but if we limited the number of branches it seems like we'd be forced to throw away data.

I agree that if the tree is memory-resident there will always be a point at which it will
crash the VM, but if we can architect things so that point only occurs when the tree itself
requires gigabytes of memory we'd be in pretty good shape.  I have the sense that the offending
tree is really only ~10 MB, but that the traversal does some N^2 explosion.  I could be wrong.

Disk-resident would be really cool if the most common interactions still only require one
lookup.

> out of memory crash when compacting document with lots of edit branches
> -----------------------------------------------------------------------
>
>                 Key: COUCHDB-888
>                 URL: https://issues.apache.org/jira/browse/COUCHDB-888
>             Project: CouchDB
>          Issue Type: Bug
>          Components: Database Core
>            Reporter: Adam Kocoloski
>         Attachments: key_tree_backtrace.txt.gz
>
>
> I have a database which will crash CouchDB if I try to compact it.  It causes beam.smp
to use all the memory on the server.  I caught it in the act one time and sorted the Erlang
processes by memory usage.  The process spawned to do the compaction turned out to be the
culprit.  I took a backtrace of the process and found that it was mapping a very large revision
tree.  I have reason to believe that the document has a large number (~1000s) of edit conflicts.
> I think part of the problem may be that the recursion in couch_key_tree:map_simple requires
each stack space for every iteration.  I'm not sure if it's possible to rewrite the algorithm
in a more memory-friendly way given the current tree structure.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message