From dev-return-11744-apmail-couchdb-dev-archive=couchdb.apache.org@couchdb.apache.org Thu Sep 16 21:35:55 2010 Return-Path: Delivered-To: apmail-couchdb-dev-archive@www.apache.org Received: (qmail 58660 invoked from network); 16 Sep 2010 21:35:55 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 16 Sep 2010 21:35:55 -0000 Received: (qmail 12383 invoked by uid 500); 16 Sep 2010 21:35:54 -0000 Delivered-To: apmail-couchdb-dev-archive@couchdb.apache.org Received: (qmail 12314 invoked by uid 500); 16 Sep 2010 21:35:53 -0000 Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@couchdb.apache.org Delivered-To: mailing list dev@couchdb.apache.org Received: (qmail 12306 invoked by uid 99); 16 Sep 2010 21:35:53 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Sep 2010 21:35:53 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Sep 2010 21:35:53 +0000 Received: from thor (localhost [127.0.0.1]) by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o8GLZWAb029894 for ; Thu, 16 Sep 2010 21:35:33 GMT Message-ID: <13971130.237981284672932432.JavaMail.jira@thor> Date: Thu, 16 Sep 2010 17:35:32 -0400 (EDT) From: "Adam Kocoloski (JIRA)" To: dev@couchdb.apache.org Subject: [jira] Commented: (COUCHDB-888) out of memory crash when compacting document with lots of edit branches In-Reply-To: <4551134.236811284669152441.JavaMail.jira@thor> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/COUCHDB-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12910328#action_12910328 ] Adam Kocoloski commented on COUCHDB-888: ---------------------------------------- Hi Damien, I guess the implications of limiting the number of edit branches would be pretty serious, right? When we stem a branch we run the risk of introducing spurious conflicts, but if we limited the number of branches it seems like we'd be forced to throw away data. I agree that if the tree is memory-resident there will always be a point at which it will crash the VM, but if we can architect things so that point only occurs when the tree itself requires gigabytes of memory we'd be in pretty good shape. I have the sense that the offending tree is really only ~10 MB, but that the traversal does some N^2 explosion. I could be wrong. Disk-resident would be really cool if the most common interactions still only require one lookup. > out of memory crash when compacting document with lots of edit branches > ----------------------------------------------------------------------- > > Key: COUCHDB-888 > URL: https://issues.apache.org/jira/browse/COUCHDB-888 > Project: CouchDB > Issue Type: Bug > Components: Database Core > Reporter: Adam Kocoloski > Attachments: key_tree_backtrace.txt.gz > > > I have a database which will crash CouchDB if I try to compact it. It causes beam.smp to use all the memory on the server. I caught it in the act one time and sorted the Erlang processes by memory usage. The process spawned to do the compaction turned out to be the culprit. I took a backtrace of the process and found that it was mapping a very large revision tree. I have reason to believe that the document has a large number (~1000s) of edit conflicts. > I think part of the problem may be that the recursion in couch_key_tree:map_simple requires each stack space for every iteration. I'm not sure if it's possible to rewrite the algorithm in a more memory-friendly way given the current tree structure. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.