From user-return-12936-apmail-couchdb-user-archive=couchdb.apache.org@couchdb.apache.org Mon Sep 27 10:15:28 2010 Return-Path: Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: (qmail 8229 invoked from network); 27 Sep 2010 10:15:28 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 27 Sep 2010 10:15:28 -0000 Received: (qmail 50681 invoked by uid 500); 27 Sep 2010 10:15:27 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 50334 invoked by uid 500); 27 Sep 2010 10:15:24 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 50325 invoked by uid 99); 27 Sep 2010 10:15:23 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Sep 2010 10:15:22 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of emargee@gmail.com designates 209.85.210.52 as permitted sender) Received: from [209.85.210.52] (HELO mail-pz0-f52.google.com) (209.85.210.52) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Sep 2010 10:15:16 +0000 Received: by pzk27 with SMTP id 27so1524444pzk.11 for ; Mon, 27 Sep 2010 03:14:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=/gTxtn+n/ZCLjMXPxjZWe2122i/2JNMVT8j/IsPZ9Bg=; b=voMNBBbQYDQZs+joclnX7QKKQ3Li0s8pinj38kOsLmYHmRmoOhOMUlNyMjVZKVVWxZ h5MyPgB8ji2mO/W2VlH516Ti6VVSLCJvEv40r5Zbol6wuMmIsJcwNLPE3orfme//3ERy 0Qi4eI2Mq7zIYnhKimgbEadqNxzdL7KvOeueM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=lx9aMCE+kV/pOvF+o4PWSYHbcOV5zQ+bT8aeAgS5gvsJqbMoaUi+fXgmKo7Yv4psYb PmCRhypnL0G51pKW274G0WtJTjjpRxEQwF7j45Yk3Xy/gbFmkQOF3Eo67n2LVVe4WyP7 yNlEuCEPPfn4G1colZxWCPfPu+hpE3QuI2dvI= MIME-Version: 1.0 Received: by 10.142.161.10 with SMTP id j10mr4457833wfe.262.1285582496204; Mon, 27 Sep 2010 03:14:56 -0700 (PDT) Received: by 10.220.184.76 with HTTP; Mon, 27 Sep 2010 03:14:56 -0700 (PDT) In-Reply-To: References: Date: Mon, 27 Sep 2010 11:14:56 +0100 Message-ID: Subject: Re: Couch not releasing deleted files From: "[mRg]" To: user@couchdb.apache.org Content-Type: multipart/alternative; boundary=000e0cd32df0043dcd04913b0064 --000e0cd32df0043dcd04913b0064 Content-Type: text/plain; charset=ISO-8859-1 Anyone still following this on here, I asked this question on the dev list and Filipe has committed a potential fix for this issue here: http://svn.apache.org/viewvc?view=revision&revision=1001196 if people are still having this issue they could try applying this, we are now doing internal testing to see if this has fixed it for our use case. Regards Stephen On 19 September 2010 00:36, Robert Newson wrote: > It's been discussed by dev's at CouchCamp but a mailing list reminder > couldn't hurt. There's some discussion about 1.1, so perhaps that's a > good thread to raise this on. > > B. > > On Sun, Sep 19, 2010 at 12:24 AM, [mRg] wrote: > > Has this been raised with the dev list or is there a bug raised for this > > issue ? Seems a few people are having this and more are probably not > seeing > > yet. > > > > On 15 September 2010 08:49, [mRg] wrote: > > > >> Yes there are 6 views (2 design docs) > >> > >> > >> On 14 September 2010 21:51, Adam Kocoloski wrote: > >> > >>> Were there any view indexes on that DB? My suspicion is that > >>> couch_view_group misses a step somewhere in its dance and keeps a > reference > >>> to an old #db{} > >>> > >>> Adam > >>> > >>> On Sep 14, 2010, at 4:46 PM, [mRg] wrote: > >>> > >>> > It was just on a DB with a lot of writes (documents having lots of > >>> > revisions, rather that lots of new docs), the database grows as > expected > >>> but > >>> > after just doing a simple compact on the db the old files still seem > to > >>> be > >>> > on disk as reported previously. > >>> > > >>> > It seems its been around a while now as (as far as we can see) 0.11 / > >>> 1.0 > >>> > and 1.0.1 all have the same issue, when we replicated the same data > over > >>> to > >>> > the newer versions the same identical problem happened. > >>> > > >>> > On 14 September 2010 14:49, Robert Newson > >>> wrote: > >>> > > >>> >> This was mentioned at CouchCamp, several developers suspect there's > a > >>> >> subtle bug in the code where file descriptors are not being closed > >>> >> under some conditions. > >>> >> > >>> >> Did the OP identify a particular sequence that led here? > >>> >> > >>> >> B. > >>> >> > >>> >> On Mon, Sep 13, 2010 at 6:25 PM, Juan Jose Comellas < > >>> juanjo@comellas.org> > >>> >> wrote: > >>> >>> I'm pretty certain it's a bug in CouchDB. I haven't checked the > code, > >>> but > >>> >>> what is happening is that CouchDB is keeping file descriptors > >>> >> corresponding > >>> >>> to deleted files open, probably after compacting a database. I'm > >>> >> inferring > >>> >>> this based on the fact that with lsof I saw several cases of two > files > >>> >> with > >>> >>> the same name (one deleted and one not) opened by CouchDB. > >>> >>> > >>> >>> > >>> >>> On Mon, Sep 13, 2010 at 7:28 PM, [mRg] wrote: > >>> >>> > >>> >>>> Is this an issue with the OS or with Couch/Beam itself ? > >>> >>>> > >>> >>>> On 10 September 2010 16:57, Juan Jose Comellas < > juanjo@comellas.org> > >>> >>>> wrote: > >>> >>>> > >>> >>>>> It also happened to me with Ubuntu 8.04 and CouchDB 0.10.x. The > only > >>> >> way > >>> >>>> I > >>> >>>>> found to solve the problem was by restarting CouchDB. > >>> >>>>> > >>> >>>>> > >>> >>>>> On Thu, Sep 9, 2010 at 10:25 AM, [mRg] > wrote: > >>> >>>>> > >>> >>>>>> [CouchDB Version : 0.11] > >>> >>>>>> [OS: RHEL 5.5] > >>> >>>>>> > >>> >>>>>> Hi all, > >>> >>>>>> > >>> >>>>>> We are seeing some odd behaviour with CouchDb not releasing > files > >>> >> after > >>> >>>>> it > >>> >>>>>> has compacted them meaning we are seeing disk space continue to > >>> >>>> increase > >>> >>>>>> until the service is restarted even though we have regular > >>> >> compaction / > >>> >>>>>> cleanup. > >>> >>>>>> > >>> >>>>>> Typical scenario : File system alert for disk space .. > >>> >>>>>> > >>> >>>>>> df says 5.5.GB used .. > >>> >>>>>> > >>> >>>>>> df -h | grep -P 'Filesystem|db' > >>> >>>>>> > >>> >>>>>> Filesystem Size Used Avail Use% Mounted on > >>> >>>>>> > >>> >>>>>> 8.0G 5.5G 2.2G 72% /var/lib/db > >>> >>>>>> > >>> >>>>>> > >>> >>>>>> .. but the files on the disk (and in Futon) say only 793MB is > being > >>> >>>> used > >>> >>>>> .. > >>> >>>>>> > >>> >>>>>> > >>> >>>>>> du -sh /var/lib/db/* > >>> >>>>>> > >>> >>>>>> 793M /var/lib/db/couchdb > >>> >>>>>> > >>> >>>>>> > >>> >>>>>> .. running lsof shows the files marked as deleted but still > there > >>> >> (I'm > >>> >>>>>> assuming beam still has some kind of handle still open to them) > .. > >>> >>>>>> > >>> >>>>>> > >>> >>>>>> lsof | grep -P 'COMMAND|/var/lib/db/couchdb/' > >>> >>>>>> > >>> >>>>>> COMMAND PID USER FD TYPE DEVICE > >>> >> SIZE > >>> >>>>>> NODE NAME > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 12u REG 253,10 > >>> >> 4183 > >>> >>>>>> 1210374 > >>> >>>>>> > >>> >>>>>> > >>> >>>>> > >>> >>>> > >>> >> > >>> > /var/lib/db/couchdb/.active_context_design/8efe9584a944f4b24b97f134f541d80f.view > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 16u REG 253,10 > >>> >> 612039080 > >>> >>>>>> 1204227 /var/lib/db/couchdb/active_context.couch (deleted) > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 20u REG 253,10 > >>> >> 180699242 > >>> >>>>>> 1204237 /var/lib/db/couchdb/active_context.couch (deleted) > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 21u REG 253,10 > >>> >> 3133540 > >>> >>>>>> 2023429 /var/lib/db/couchdb/thesaurus.couch (deleted) > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 25u REG 253,10 > >>> >> 71422052 > >>> >>>>>> 2023428 /var/lib/db/couchdb/active_context_schedule.couch > (deleted) > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 26u REG 253,10 > >>> >> 71372900 > >>> >>>>>> 2056193 /var/lib/db/couchdb/active_context_schedule.couch > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 28u REG 253,10 > >>> >> 621252717 > >>> >>>>>> 1204238 /var/lib/db/couchdb/active_context.couch > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 29u REG 253,10 > >>> >> 1740903 > >>> >>>>>> 90113 /var/lib/db/couchdb/mapping_cache.couch > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 30u REG 253,10 > >>> >> 3133540 > >>> >>>>>> 1204253 /var/lib/db/couchdb/thesaurus.couch (deleted) > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 31u REG 253,10 > >>> >> 2318439 > >>> >>>>>> 1204240 /var/lib/db/couchdb/mappings.couch (deleted) > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 32u REG 253,10 > >>> >> 983150 > >>> >>>>>> 2004994 > >>> >>>>>> > >>> >>>>> > >>> >>>> > >>> >> > >>> > /var/lib/db/couchdb/.mappings_design/f9964c50b95c167c8ac3d8d3b4c76a4b.view > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 33u REG 253,10 > >>> >> 3133540 > >>> >>>>>> 2056194 /var/lib/db/couchdb/thesaurus.couch > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 34u REG 253,10 > >>> >> 70742116 > >>> >>>>>> 1204231 /var/lib/db/couchdb/active_context_schedule.couch > (deleted) > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 44u REG 253,10 > >>> >> 1740903 > >>> >>>>>> 2023427 /var/lib/db/couchdb/mapping_cache.couch (deleted) > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 45u REG 253,10 > >>> >> 65396836 > >>> >>>>>> 1994756 /var/lib/db/couchdb/active_context_schedule.couch > (deleted) > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 47u REG 253,10 > >>> >> 71229540 > >>> >>>>>> 1204243 /var/lib/db/couchdb/active_context_schedule.couch > (deleted) > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 48u REG 253,10 > >>> >> 240254078 > >>> >>>>>> 1204226 /var/lib/db/couchdb/active_context.couch (deleted) > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 49u REG 253,10 > >>> >> 2232423 > >>> >>>>>> 1204241 /var/lib/db/couchdb/mappings.couch > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 51u REG 253,10 > >>> >> 96903353 > >>> >>>>>> 1210371 > >>> >>>>>> > >>> >>>>>> > >>> >>>>> > >>> >>>> > >>> >> > >>> > /var/lib/db/couchdb/.active_context_design/b406f7bb6d01b6d50324f14cde588324.view > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 52u REG 253,10 > >>> >> 1192034 > >>> >>>>>> 1249282 > >>> >>>>>> > >>> >>>>>> > >>> >>>>> > >>> >>>> > >>> >> > >>> > /var/lib/db/couchdb/.mapping_cache_design/8bbf80a6e1ca8f2c71ebc90c730e7190.view > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 53u REG 253,10 > >>> >> 1622135 > >>> >>>>>> 2004993 > >>> >>>>>> > >>> >>>>> > >>> >>>> > >>> >> > >>> > /var/lib/db/couchdb/.mappings_design/1ff69004b575837aae9d4147162197d4.view > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 54u REG 253,10 > >>> >> 4172 > >>> >>>>>> 1208327 > >>> >>>>>> > >>> >>>>> > >>> >>>> > >>> >> > >>> > /var/lib/db/couchdb/.thesaurus_design/73fe79b20ff06e341257fa78231ca87e.view > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 55u REG 253,10 > >>> >> 13496483 > >>> >>>>>> 1208325 > >>> >>>>>> > >>> >>>>> > >>> >>>> > >>> >> > >>> > /var/lib/db/couchdb/.thesaurus_design/fcc08d6e41c54c0d470c5984ea9eb6e3.view > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 56u REG 253,10 > >>> >> 12959902 > >>> >>>>>> 1208326 > >>> >>>>>> > >>> >>>>> > >>> >>>> > >>> >> > >>> > /var/lib/db/couchdb/.thesaurus_design/c06195da19f34ced37fa1ab91e663c73.view > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 57u REG 253,10 > >>> >> 70254692 > >>> >>>>>> 1204236 /var/lib/db/couchdb/active_context_schedule.couch > (deleted) > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 58u REG 253,10 > >>> >> 581730 > >>> >>>>>> 1210373 > >>> >>>>>> > >>> >>>>>> > >>> >>>>> > >>> >>>> > >>> >> > >>> > /var/lib/db/couchdb/.active_context_schedule_design/c64c047dcff74ea2122356efb1042759.view > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 67u REG 253,10 > >>> >> 1818727 > >>> >>>>>> 1204247 /var/lib/db/couchdb/mapping_cache.couch (deleted) > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 78u REG 253,10 > >>> >> 609905146 > >>> >>>>>> 1204234 /var/lib/db/couchdb/active_context.couch (deleted) > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 80u REG 253,10 > >>> >> 545919085 > >>> >>>>>> 1994753 /var/lib/db/couchdb/active_context.couch (deleted) > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 81u REG 253,10 > >>> >> 71032932 > >>> >>>>>> 1204249 /var/lib/db/couchdb/active_context_schedule.couch > (deleted) > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 84u REG 253,10 > >>> >> 548866111 > >>> >>>>>> 1204239 /var/lib/db/couchdb/active_context.couch (deleted) > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 85u REG 253,10 > >>> >> 564612402 > >>> >>>>>> 1204233 /var/lib/db/couchdb/active_context.couch (deleted) > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 87u REG 253,10 > >>> >> 615996179 > >>> >>>>>> 1204228 /var/lib/db/couchdb/active_context.couch (deleted) > >>> >>>>>> > >>> >>>>>> beam.smp 21352 couchdb 102u REG 253,10 > >>> >> 561111149 > >>> >>>>>> 2023425 /var/lib/db/couchdb/active_context.couch (deleted) > >>> >>>>>> > >>> >>>>>> > >>> >>>>>> Has anyone seen this behaviour before or knows of a way around > this > >>> >> ? > >>> >>>>>> Currently we have to restart the couchdb service in order to > >>> restore > >>> >>>> the > >>> >>>>>> disk space. > >>> >>>>>> > >>> >>>>>> Any help/advice is much appreciated. > >>> >>>>>> > >>> >>>>>> Regards > >>> >>>>>> > >>> >>>>>> Stephen > >>> >>>>>> > >>> >>>>> > >>> >>>> > >>> >>> > >>> >> > >>> > >>> > >> > > > --000e0cd32df0043dcd04913b0064--