From dev-return-17746-apmail-couchdb-dev-archive=couchdb.apache.org@couchdb.apache.org Wed Aug 24 07:46:09 2011 Return-Path: X-Original-To: apmail-couchdb-dev-archive@www.apache.org Delivered-To: apmail-couchdb-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A16028765 for ; Wed, 24 Aug 2011 07:46:09 +0000 (UTC) Received: (qmail 61790 invoked by uid 500); 24 Aug 2011 07:46:09 -0000 Delivered-To: apmail-couchdb-dev-archive@couchdb.apache.org Received: (qmail 60878 invoked by uid 500); 24 Aug 2011 07:46:00 -0000 Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@couchdb.apache.org Delivered-To: mailing list dev@couchdb.apache.org Received: (qmail 60865 invoked by uid 99); 24 Aug 2011 07:45:55 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Aug 2011 07:45:55 +0000 X-ASF-Spam-Status: No, hits=-2000.9 required=5.0 tests=ALL_TRUSTED,RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Aug 2011 07:45:52 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116]) by hel.zones.apache.org (Postfix) with ESMTP id 0803ACCFBB for ; Wed, 24 Aug 2011 07:45:29 +0000 (UTC) Date: Wed, 24 Aug 2011 07:45:29 +0000 (UTC) From: "Randall Leeds (JIRA)" To: dev@couchdb.apache.org Message-ID: <57433269.8573.1314171929029.JavaMail.tomcat@hel.zones.apache.org> In-Reply-To: <743009926.4877.1314098069637.JavaMail.tomcat@hel.zones.apache.org> Subject: [jira] [Updated] (COUCHDB-1258) eheap_alloc OOM errors when attempting to build selected views MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/COUCHDB-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randall Leeds updated COUCHDB-1258: ----------------------------------- Affects Version/s: 1.1 Surely one of these two views emits many more documents than the other and there is something about that which is exhausting heap space. The negation in JavaScript cannot affect the memory usage of BEAM because the view server runs in a separate process with its own heap and the errors you see are from Erlang. Someone will have to take a deep look into the view code and audit it for any sloppy memory usage. As an experiment could you try emitting null as the value rather than doc? If the docs are very big perhaps we need to flush them in smaller batches or something. > eheap_alloc OOM errors when attempting to build selected views > -------------------------------------------------------------- > > Key: COUCHDB-1258 > URL: https://issues.apache.org/jira/browse/COUCHDB-1258 > Project: CouchDB > Issue Type: Bug > Affects Versions: 1.1 > Environment: CentOS 5.6, 1GB RAM (approx 800MB free) > CouchDb 1.1.0, Erlang R14B-03, js lib 1.7.0-8 EPEL RPM build > Couch database exhibiting this behaviour: > {"db_name":"activity_new","doc_count":593274,"doc_del_count":4743352,"update_seq":10559287,"purge_seq":0,"compact_running":false,"disk_size":3366396013,"instance_start_time":"1314097423985726","disk_format_version":5,"committed_update_seq":10559287} > Reporter: James Cohen > Priority: Critical > > We spotted OOM errors crashing our CouchDb instance when attempting to rebuild selected views. CouchDb was dying with the following messages (worth noting that the type reported varies between heap/old_heap > eheap_alloc: Cannot allocate 478288480 bytes of memory (of type "heap"). > eheap_alloc: Cannot allocate 597860600 bytes of memory (of type "heap"). > eheap_alloc: Cannot allocate 747325720 bytes of memory (of type "old_heap"). > eheap_alloc: Cannot allocate 597860600 bytes of memory (of type "old_heap"). > By modifying the view I was able to find a view that could consistently crash the server and another that ran fine. They are as follows: > Runs out of memory v.quickly > { > "_id": "_design/cleanup", > "_rev": "5-e004fbab278355e9d08763877e5a8295", > "views": { > "byDate": { > "map": "function(doc) { if (! doc.action) emit([doc.date], doc); }" > } > } > } > Runs fine with minimal memory usage (returns 88128 docs in the view) > { > "_id": "_design/cleanup", > "_rev": "6-3823be6b72ca2441e235addfece6900c", > "views": { > "byDate": { > "map": "function(doc) { if (doc.action) emit([doc.date], doc); }" > } > } > } > The only difference between the two is the negation of the if conditional. > memory usage was monitored with top on the machine while the view was being built. Under correct behaviour I could see beam.smp using just 3 or 4% of the server's memory. With the view that causes problems that memory usage increased until the RAM/swap on the server was exhausted (as you can see from the error messages around 500/700MB) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira