couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Newson <robert.new...@gmail.com>
Subject Re: Couch not releasing deleted files
Date Tue, 14 Sep 2010 13:49:58 GMT
This was mentioned at CouchCamp, several developers suspect there's a
subtle bug in the code where file descriptors are not being closed
under some conditions.

Did the OP identify a particular sequence that led here?

B.

On Mon, Sep 13, 2010 at 6:25 PM, Juan Jose Comellas <juanjo@comellas.org> wrote:
> I'm pretty certain it's a bug in CouchDB. I haven't checked the code, but
> what is happening is that CouchDB is keeping file descriptors corresponding
> to deleted files open, probably after compacting a database. I'm inferring
> this based on the fact that with lsof I saw several cases of two files with
> the same name (one deleted and one not) opened by CouchDB.
>
>
> On Mon, Sep 13, 2010 at 7:28 PM, [mRg] <emargee@gmail.com> wrote:
>
>> Is this an issue with the OS or with Couch/Beam itself ?
>>
>> On 10 September 2010 16:57, Juan Jose Comellas <juanjo@comellas.org>
>> wrote:
>>
>> > It also happened to me with Ubuntu 8.04 and CouchDB 0.10.x. The only way
>> I
>> > found to solve the problem was by restarting CouchDB.
>> >
>> >
>> > On Thu, Sep 9, 2010 at 10:25 AM, [mRg] <emargee@gmail.com> wrote:
>> >
>> > > [CouchDB Version : 0.11]
>> > > [OS: RHEL 5.5]
>> > >
>> > > Hi all,
>> > >
>> > > We are seeing some odd behaviour with CouchDb not releasing files after
>> > it
>> > > has compacted them meaning we are seeing disk space continue to
>> increase
>> > > until the service is restarted even though we have regular compaction /
>> > > cleanup.
>> > >
>> > > Typical scenario : File system alert for disk space ..
>> > >
>> > > df says 5.5.GB used ..
>> > >
>> > > df -h | grep -P 'Filesystem|db'
>> > >
>> > > Filesystem            Size  Used Avail Use% Mounted on
>> > >
>> > >                      8.0G  5.5G  2.2G  72% /var/lib/db
>> > >
>> > >
>> > > .. but the files on the disk (and in Futon) say only 793MB is being
>> used
>> > ..
>> > >
>> > >
>> > > du -sh /var/lib/db/*
>> > >
>> > > 793M     /var/lib/db/couchdb
>> > >
>> > >
>> > > .. running lsof shows the files marked as deleted but still there (I'm
>> > > assuming beam still has some kind of handle still open to them) ..
>> > >
>> > >
>> > > lsof | grep -P 'COMMAND|/var/lib/db/couchdb/'
>> > >
>> > > COMMAND     PID      USER   FD      TYPE             DEVICE
     SIZE
>> > > NODE NAME
>> > >
>> > > beam.smp  21352   couchdb   12u      REG             253,10
     4183
>> > > 1210374
>> > >
>> > >
>> >
>> /var/lib/db/couchdb/.active_context_design/8efe9584a944f4b24b97f134f541d80f.view
>> > >
>> > > beam.smp  21352   couchdb   16u      REG             253,10
612039080
>> > > 1204227 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >
>> > > beam.smp  21352   couchdb   20u      REG             253,10
180699242
>> > > 1204237 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >
>> > > beam.smp  21352   couchdb   21u      REG             253,10
  3133540
>> > > 2023429 /var/lib/db/couchdb/thesaurus.couch (deleted)
>> > >
>> > > beam.smp  21352   couchdb   25u      REG             253,10
 71422052
>> > > 2023428 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>> > >
>> > > beam.smp  21352   couchdb   26u      REG             253,10
 71372900
>> > > 2056193 /var/lib/db/couchdb/active_context_schedule.couch
>> > >
>> > > beam.smp  21352   couchdb   28u      REG             253,10
621252717
>> > > 1204238 /var/lib/db/couchdb/active_context.couch
>> > >
>> > > beam.smp  21352   couchdb   29u      REG             253,10
  1740903
>> > > 90113 /var/lib/db/couchdb/mapping_cache.couch
>> > >
>> > > beam.smp  21352   couchdb   30u      REG             253,10
  3133540
>> > > 1204253 /var/lib/db/couchdb/thesaurus.couch (deleted)
>> > >
>> > > beam.smp  21352   couchdb   31u      REG             253,10
  2318439
>> > > 1204240 /var/lib/db/couchdb/mappings.couch (deleted)
>> > >
>> > > beam.smp  21352   couchdb   32u      REG             253,10
   983150
>> > > 2004994
>> > >
>> >
>> /var/lib/db/couchdb/.mappings_design/f9964c50b95c167c8ac3d8d3b4c76a4b.view
>> > >
>> > > beam.smp  21352   couchdb   33u      REG             253,10
  3133540
>> > > 2056194 /var/lib/db/couchdb/thesaurus.couch
>> > >
>> > > beam.smp  21352   couchdb   34u      REG             253,10
 70742116
>> > > 1204231 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>> > >
>> > > beam.smp  21352   couchdb   44u      REG             253,10
  1740903
>> > > 2023427 /var/lib/db/couchdb/mapping_cache.couch (deleted)
>> > >
>> > > beam.smp  21352   couchdb   45u      REG             253,10
 65396836
>> > > 1994756 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>> > >
>> > > beam.smp  21352   couchdb   47u      REG             253,10
 71229540
>> > > 1204243 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>> > >
>> > > beam.smp  21352   couchdb   48u      REG             253,10
240254078
>> > >   1204226 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >
>> > > beam.smp  21352   couchdb   49u      REG             253,10
  2232423
>> > > 1204241 /var/lib/db/couchdb/mappings.couch
>> > >
>> > > beam.smp  21352   couchdb   51u      REG             253,10
 96903353
>> > > 1210371
>> > >
>> > >
>> >
>> /var/lib/db/couchdb/.active_context_design/b406f7bb6d01b6d50324f14cde588324.view
>> > >
>> > > beam.smp  21352   couchdb   52u      REG             253,10
  1192034
>> > > 1249282
>> > >
>> > >
>> >
>> /var/lib/db/couchdb/.mapping_cache_design/8bbf80a6e1ca8f2c71ebc90c730e7190.view
>> > >
>> > > beam.smp  21352   couchdb   53u      REG             253,10
  1622135
>> > > 2004993
>> > >
>> >
>> /var/lib/db/couchdb/.mappings_design/1ff69004b575837aae9d4147162197d4.view
>> > >
>> > > beam.smp  21352   couchdb   54u      REG             253,10
     4172
>> > > 1208327
>> > >
>> >
>> /var/lib/db/couchdb/.thesaurus_design/73fe79b20ff06e341257fa78231ca87e.view
>> > >
>> > > beam.smp  21352   couchdb   55u      REG             253,10
 13496483
>> > > 1208325
>> > >
>> >
>> /var/lib/db/couchdb/.thesaurus_design/fcc08d6e41c54c0d470c5984ea9eb6e3.view
>> > >
>> > > beam.smp  21352   couchdb   56u      REG             253,10
 12959902
>> > > 1208326
>> > >
>> >
>> /var/lib/db/couchdb/.thesaurus_design/c06195da19f34ced37fa1ab91e663c73.view
>> > >
>> > > beam.smp  21352   couchdb   57u      REG             253,10
 70254692
>> > > 1204236 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>> > >
>> > > beam.smp  21352   couchdb   58u      REG             253,10
   581730
>> > > 1210373
>> > >
>> > >
>> >
>> /var/lib/db/couchdb/.active_context_schedule_design/c64c047dcff74ea2122356efb1042759.view
>> > >
>> > > beam.smp  21352   couchdb   67u      REG             253,10
  1818727
>> > > 1204247 /var/lib/db/couchdb/mapping_cache.couch (deleted)
>> > >
>> > > beam.smp  21352   couchdb   78u      REG             253,10
609905146
>> > > 1204234 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >
>> > > beam.smp  21352   couchdb   80u      REG             253,10
545919085
>> > > 1994753 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >
>> > > beam.smp  21352   couchdb   81u      REG             253,10
 71032932
>> > > 1204249 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>> > >
>> > > beam.smp  21352   couchdb   84u      REG             253,10
548866111
>> > > 1204239 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >
>> > > beam.smp  21352   couchdb   85u      REG             253,10
564612402
>> > > 1204233 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >
>> > > beam.smp  21352   couchdb   87u      REG             253,10
615996179
>> > > 1204228 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >
>> > > beam.smp  21352   couchdb  102u      REG             253,10
561111149
>> > > 2023425 /var/lib/db/couchdb/active_context.couch (deleted)
>> > >
>> > >
>> > > Has anyone seen this behaviour before or knows of a way around this ?
>> > > Currently we have to restart the couchdb service in order to restore
>> the
>> > > disk space.
>> > >
>> > > Any help/advice is much appreciated.
>> > >
>> > > Regards
>> > >
>> > > Stephen
>> > >
>> >
>>
>

Mime
View raw message