Return-Path: X-Original-To: apmail-couchdb-dev-archive@www.apache.org Delivered-To: apmail-couchdb-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5B23710E03 for ; Wed, 12 Feb 2014 18:59:43 +0000 (UTC) Received: (qmail 15118 invoked by uid 500); 12 Feb 2014 18:59:31 -0000 Delivered-To: apmail-couchdb-dev-archive@couchdb.apache.org Received: (qmail 15004 invoked by uid 500); 12 Feb 2014 18:59:29 -0000 Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@couchdb.apache.org Delivered-To: mailing list dev@couchdb.apache.org Received: (qmail 14741 invoked by uid 99); 12 Feb 2014 18:59:25 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Feb 2014 18:59:25 +0000 Date: Wed, 12 Feb 2014 18:59:25 +0000 (UTC) From: "Rohit Sharma (JIRA)" To: dev@couchdb.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (COUCHDB-2058) CouchDB Memory Leak - Beam.smp MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/COUCHDB-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899433#comment-13899433 ] Rohit Sharma commented on COUCHDB-2058: --------------------------------------- Hey Paul, sorry about the confusion. I just finished another test and looks like vast amount of memory is released when I restart the CouchDB: Here are the stats: Before running process: (After rebooting the server) Mem: 8061508k total, 1694180k used, 6367328k free, 35224k buffers Swap: 2064376k total, 0k used, 2064376k free, 290184k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 11671 root 20 0 591m 77m 4372 S 1.3 1.0 0:07.63 /usr/local/lib/erlang/erts-5.10.4/bin/beam.smp -Bd -K true -A 4 -- -root /usr/local/lib/erlang -progname erl -- -home /root -- -nosh After Finishing the process - 7.2G (1.8G is page cache) memory consumed out of 8G Mem: 8061508k total, 7212840k used, 848668k free, 38788k buffers Swap: 2064376k total, 0k used, 2064376k free, 1836020k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 11671 root 20 0 4653m 3.8g 4392 S 2.0 48.8 41:41.53 /usr/local/lib/erlang/erts-5.10.4/bin/beam.smp -Bd -K true -A 4 -- -root /usr/local/lib/erlang -progname erl -- -home /root -- -nosh After restarting CoucchDB (4G of memory released by CouchDB) Mem: 8061508k total, 3297380k used, 4764128k free, 38940k buffers Swap: 2064376k total, 0k used, 2064376k free, 1837240k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12305 root 20 0 532m 24m 4212 S 0.3 0.3 0:00.98 /usr/local/lib/erlang/erts-5.10.4/bin/beam.smp -Bd -K true -A 4 -- -root /usr/local/lib/erlang -progname erl -- -home /root -- -nosh Could you please share your thoughts on these stats. Appreciated! > CouchDB Memory Leak - Beam.smp > ------------------------------ > > Key: COUCHDB-2058 > URL: https://issues.apache.org/jira/browse/COUCHDB-2058 > Project: CouchDB > Issue Type: Bug > Security Level: public(Regular issues) > Components: Database Core > Reporter: Rohit Sharma > > Hello, > I am experiencing performance issue with CouchDB. > Use Case: I am working on a process that retrieves the data from RDBMS and process them into JSON document and POST them to the CouchDB. > I am trying to POST around half a million documents, most of them in batches (_bulk_doc) of 10,000 and have tried with batch of 5,000, 15,000, and 20,000. > Whole process takes around 90-100 minutes. > During the life of the process, Memory Consumption by CouchDB keeps on growing and memory is not released when CouchDB has finished working. > So if the memory consumption by CouchDB was 60% at the time process finishes, memory consumption will remain 60% and not reducing. > Subsequently, when the process starts running again. memory consumption is Maxed out and CouchDB restarts itself. This restart fails the process that I am running. Looking at the Syslogs , I see Out Of Memory Error by the CouchDB process and killing statement. > The CouchDb process that has the issue is the "beam.smp" of Erlang. > At this point, I have tried upgrading the memory of the server to see if this resolves the issue, unfortunately, the issue persists. Memory Leak is there and Usage keeps on growing until CouchDB restarts/crashed. > I also have tried running garbage collection from Erlang command (erlang:garbage_collect().) line but it didn't do anything. > At this point, I am out of ideas and not sure what is going on here. Any input/suggestion is highly appreciated! > Env: > Platform: Linux (Red Hat release 6.4 (Santiago)) > CouchDB: 1.3 and have tried with 1.5 as well > RAM: Tried with 2G, 4G, and 8G > CPU: 2 cores > Process:/usr/lib64/erlang/erts-5.8.5/bin/beam.smp -Bd -K true -A 4 -- -root /usr/lib64/erlang -- This message was sent by Atlassian JIRA (v6.1.5#6160)