Return-Path: Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: (qmail 37976 invoked from network); 18 Sep 2010 23:25:26 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 18 Sep 2010 23:25:26 -0000 Received: (qmail 97874 invoked by uid 500); 18 Sep 2010 23:25:25 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 97821 invoked by uid 500); 18 Sep 2010 23:25:24 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 97813 invoked by uid 99); 18 Sep 2010 23:25:24 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Sep 2010 23:25:24 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of emargee@gmail.com designates 209.85.212.52 as permitted sender) Received: from [209.85.212.52] (HELO mail-vw0-f52.google.com) (209.85.212.52) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Sep 2010 23:25:18 +0000 Received: by vws14 with SMTP id 14so4061032vws.11 for ; Sat, 18 Sep 2010 16:24:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=mgis8f1pHA+NSxQA/DF0Ns6bs5gNOK2/L0XHmQHa2Nw=; b=xwE08T4/FIie0muwpncV/yKMaDxwfcTClgZCdqi2mcQah/DVMV30juPQp3uXId1zch 4XNi9ba4/i1puSFUrvApQqNS6IGCJBD1pjGNseJz3eZXoBUG+JpmCgAxSeqZ4UJpmxmP Di5n1rpC4hhlnjcOvY4KM31sUI/YaKcYJD4DY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=ePdWP/14CxGBc/nhbbrDU2wjLQFxZp/7kFApTtOI+Ia43PgBXce7VWHv0HFfO2RZ8+ T9DQHaS8h5kFQd7t4ES9GsFbDY5Ov2HeuiMeIvUsQ1FeHklr0VtUnWYtb8wKOloekT+c az+fWdQpxexVrfgpIy5keRYAOPzH2CgTbHDOQ= MIME-Version: 1.0 Received: by 10.220.123.72 with SMTP id o8mr3595762vcr.238.1284852297010; Sat, 18 Sep 2010 16:24:57 -0700 (PDT) Received: by 10.220.192.65 with HTTP; Sat, 18 Sep 2010 16:24:56 -0700 (PDT) In-Reply-To: References: Date: Sun, 19 Sep 2010 00:24:56 +0100 Message-ID: Subject: Re: Couch not releasing deleted files From: "[mRg]" To: user@couchdb.apache.org Content-Type: multipart/alternative; boundary=0016e6d260b6c0942d049090fc25 --0016e6d260b6c0942d049090fc25 Content-Type: text/plain; charset=ISO-8859-1 Has this been raised with the dev list or is there a bug raised for this issue ? Seems a few people are having this and more are probably not seeing yet. On 15 September 2010 08:49, [mRg] wrote: > Yes there are 6 views (2 design docs) > > > On 14 September 2010 21:51, Adam Kocoloski wrote: > >> Were there any view indexes on that DB? My suspicion is that >> couch_view_group misses a step somewhere in its dance and keeps a reference >> to an old #db{} >> >> Adam >> >> On Sep 14, 2010, at 4:46 PM, [mRg] wrote: >> >> > It was just on a DB with a lot of writes (documents having lots of >> > revisions, rather that lots of new docs), the database grows as expected >> but >> > after just doing a simple compact on the db the old files still seem to >> be >> > on disk as reported previously. >> > >> > It seems its been around a while now as (as far as we can see) 0.11 / >> 1.0 >> > and 1.0.1 all have the same issue, when we replicated the same data over >> to >> > the newer versions the same identical problem happened. >> > >> > On 14 September 2010 14:49, Robert Newson >> wrote: >> > >> >> This was mentioned at CouchCamp, several developers suspect there's a >> >> subtle bug in the code where file descriptors are not being closed >> >> under some conditions. >> >> >> >> Did the OP identify a particular sequence that led here? >> >> >> >> B. >> >> >> >> On Mon, Sep 13, 2010 at 6:25 PM, Juan Jose Comellas < >> juanjo@comellas.org> >> >> wrote: >> >>> I'm pretty certain it's a bug in CouchDB. I haven't checked the code, >> but >> >>> what is happening is that CouchDB is keeping file descriptors >> >> corresponding >> >>> to deleted files open, probably after compacting a database. I'm >> >> inferring >> >>> this based on the fact that with lsof I saw several cases of two files >> >> with >> >>> the same name (one deleted and one not) opened by CouchDB. >> >>> >> >>> >> >>> On Mon, Sep 13, 2010 at 7:28 PM, [mRg] wrote: >> >>> >> >>>> Is this an issue with the OS or with Couch/Beam itself ? >> >>>> >> >>>> On 10 September 2010 16:57, Juan Jose Comellas >> >>>> wrote: >> >>>> >> >>>>> It also happened to me with Ubuntu 8.04 and CouchDB 0.10.x. The only >> >> way >> >>>> I >> >>>>> found to solve the problem was by restarting CouchDB. >> >>>>> >> >>>>> >> >>>>> On Thu, Sep 9, 2010 at 10:25 AM, [mRg] wrote: >> >>>>> >> >>>>>> [CouchDB Version : 0.11] >> >>>>>> [OS: RHEL 5.5] >> >>>>>> >> >>>>>> Hi all, >> >>>>>> >> >>>>>> We are seeing some odd behaviour with CouchDb not releasing files >> >> after >> >>>>> it >> >>>>>> has compacted them meaning we are seeing disk space continue to >> >>>> increase >> >>>>>> until the service is restarted even though we have regular >> >> compaction / >> >>>>>> cleanup. >> >>>>>> >> >>>>>> Typical scenario : File system alert for disk space .. >> >>>>>> >> >>>>>> df says 5.5.GB used .. >> >>>>>> >> >>>>>> df -h | grep -P 'Filesystem|db' >> >>>>>> >> >>>>>> Filesystem Size Used Avail Use% Mounted on >> >>>>>> >> >>>>>> 8.0G 5.5G 2.2G 72% /var/lib/db >> >>>>>> >> >>>>>> >> >>>>>> .. but the files on the disk (and in Futon) say only 793MB is being >> >>>> used >> >>>>> .. >> >>>>>> >> >>>>>> >> >>>>>> du -sh /var/lib/db/* >> >>>>>> >> >>>>>> 793M /var/lib/db/couchdb >> >>>>>> >> >>>>>> >> >>>>>> .. running lsof shows the files marked as deleted but still there >> >> (I'm >> >>>>>> assuming beam still has some kind of handle still open to them) .. >> >>>>>> >> >>>>>> >> >>>>>> lsof | grep -P 'COMMAND|/var/lib/db/couchdb/' >> >>>>>> >> >>>>>> COMMAND PID USER FD TYPE DEVICE >> >> SIZE >> >>>>>> NODE NAME >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 12u REG 253,10 >> >> 4183 >> >>>>>> 1210374 >> >>>>>> >> >>>>>> >> >>>>> >> >>>> >> >> >> /var/lib/db/couchdb/.active_context_design/8efe9584a944f4b24b97f134f541d80f.view >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 16u REG 253,10 >> >> 612039080 >> >>>>>> 1204227 /var/lib/db/couchdb/active_context.couch (deleted) >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 20u REG 253,10 >> >> 180699242 >> >>>>>> 1204237 /var/lib/db/couchdb/active_context.couch (deleted) >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 21u REG 253,10 >> >> 3133540 >> >>>>>> 2023429 /var/lib/db/couchdb/thesaurus.couch (deleted) >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 25u REG 253,10 >> >> 71422052 >> >>>>>> 2023428 /var/lib/db/couchdb/active_context_schedule.couch (deleted) >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 26u REG 253,10 >> >> 71372900 >> >>>>>> 2056193 /var/lib/db/couchdb/active_context_schedule.couch >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 28u REG 253,10 >> >> 621252717 >> >>>>>> 1204238 /var/lib/db/couchdb/active_context.couch >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 29u REG 253,10 >> >> 1740903 >> >>>>>> 90113 /var/lib/db/couchdb/mapping_cache.couch >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 30u REG 253,10 >> >> 3133540 >> >>>>>> 1204253 /var/lib/db/couchdb/thesaurus.couch (deleted) >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 31u REG 253,10 >> >> 2318439 >> >>>>>> 1204240 /var/lib/db/couchdb/mappings.couch (deleted) >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 32u REG 253,10 >> >> 983150 >> >>>>>> 2004994 >> >>>>>> >> >>>>> >> >>>> >> >> >> /var/lib/db/couchdb/.mappings_design/f9964c50b95c167c8ac3d8d3b4c76a4b.view >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 33u REG 253,10 >> >> 3133540 >> >>>>>> 2056194 /var/lib/db/couchdb/thesaurus.couch >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 34u REG 253,10 >> >> 70742116 >> >>>>>> 1204231 /var/lib/db/couchdb/active_context_schedule.couch (deleted) >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 44u REG 253,10 >> >> 1740903 >> >>>>>> 2023427 /var/lib/db/couchdb/mapping_cache.couch (deleted) >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 45u REG 253,10 >> >> 65396836 >> >>>>>> 1994756 /var/lib/db/couchdb/active_context_schedule.couch (deleted) >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 47u REG 253,10 >> >> 71229540 >> >>>>>> 1204243 /var/lib/db/couchdb/active_context_schedule.couch (deleted) >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 48u REG 253,10 >> >> 240254078 >> >>>>>> 1204226 /var/lib/db/couchdb/active_context.couch (deleted) >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 49u REG 253,10 >> >> 2232423 >> >>>>>> 1204241 /var/lib/db/couchdb/mappings.couch >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 51u REG 253,10 >> >> 96903353 >> >>>>>> 1210371 >> >>>>>> >> >>>>>> >> >>>>> >> >>>> >> >> >> /var/lib/db/couchdb/.active_context_design/b406f7bb6d01b6d50324f14cde588324.view >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 52u REG 253,10 >> >> 1192034 >> >>>>>> 1249282 >> >>>>>> >> >>>>>> >> >>>>> >> >>>> >> >> >> /var/lib/db/couchdb/.mapping_cache_design/8bbf80a6e1ca8f2c71ebc90c730e7190.view >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 53u REG 253,10 >> >> 1622135 >> >>>>>> 2004993 >> >>>>>> >> >>>>> >> >>>> >> >> >> /var/lib/db/couchdb/.mappings_design/1ff69004b575837aae9d4147162197d4.view >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 54u REG 253,10 >> >> 4172 >> >>>>>> 1208327 >> >>>>>> >> >>>>> >> >>>> >> >> >> /var/lib/db/couchdb/.thesaurus_design/73fe79b20ff06e341257fa78231ca87e.view >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 55u REG 253,10 >> >> 13496483 >> >>>>>> 1208325 >> >>>>>> >> >>>>> >> >>>> >> >> >> /var/lib/db/couchdb/.thesaurus_design/fcc08d6e41c54c0d470c5984ea9eb6e3.view >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 56u REG 253,10 >> >> 12959902 >> >>>>>> 1208326 >> >>>>>> >> >>>>> >> >>>> >> >> >> /var/lib/db/couchdb/.thesaurus_design/c06195da19f34ced37fa1ab91e663c73.view >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 57u REG 253,10 >> >> 70254692 >> >>>>>> 1204236 /var/lib/db/couchdb/active_context_schedule.couch (deleted) >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 58u REG 253,10 >> >> 581730 >> >>>>>> 1210373 >> >>>>>> >> >>>>>> >> >>>>> >> >>>> >> >> >> /var/lib/db/couchdb/.active_context_schedule_design/c64c047dcff74ea2122356efb1042759.view >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 67u REG 253,10 >> >> 1818727 >> >>>>>> 1204247 /var/lib/db/couchdb/mapping_cache.couch (deleted) >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 78u REG 253,10 >> >> 609905146 >> >>>>>> 1204234 /var/lib/db/couchdb/active_context.couch (deleted) >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 80u REG 253,10 >> >> 545919085 >> >>>>>> 1994753 /var/lib/db/couchdb/active_context.couch (deleted) >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 81u REG 253,10 >> >> 71032932 >> >>>>>> 1204249 /var/lib/db/couchdb/active_context_schedule.couch (deleted) >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 84u REG 253,10 >> >> 548866111 >> >>>>>> 1204239 /var/lib/db/couchdb/active_context.couch (deleted) >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 85u REG 253,10 >> >> 564612402 >> >>>>>> 1204233 /var/lib/db/couchdb/active_context.couch (deleted) >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 87u REG 253,10 >> >> 615996179 >> >>>>>> 1204228 /var/lib/db/couchdb/active_context.couch (deleted) >> >>>>>> >> >>>>>> beam.smp 21352 couchdb 102u REG 253,10 >> >> 561111149 >> >>>>>> 2023425 /var/lib/db/couchdb/active_context.couch (deleted) >> >>>>>> >> >>>>>> >> >>>>>> Has anyone seen this behaviour before or knows of a way around this >> >> ? >> >>>>>> Currently we have to restart the couchdb service in order to >> restore >> >>>> the >> >>>>>> disk space. >> >>>>>> >> >>>>>> Any help/advice is much appreciated. >> >>>>>> >> >>>>>> Regards >> >>>>>> >> >>>>>> Stephen >> >>>>>> >> >>>>> >> >>>> >> >>> >> >> >> >> > --0016e6d260b6c0942d049090fc25--