incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Newson <robert.new...@gmail.com>
Subject Re: Couch not releasing deleted files
Date Sat, 18 Sep 2010 23:36:26 GMT
It's been discussed by dev's at CouchCamp but a mailing list reminder
couldn't hurt. There's some discussion about 1.1, so perhaps that's a
good thread to raise this on.

B.

On Sun, Sep 19, 2010 at 12:24 AM, [mRg] <emargee@gmail.com> wrote:
> Has this been raised with the dev list or is there a bug raised for this
> issue ? Seems a few people are having this and more are probably not seeing
> yet.
>
> On 15 September 2010 08:49, [mRg] <emargee@gmail.com> wrote:
>
>> Yes there are 6 views (2 design docs)
>>
>>
>> On 14 September 2010 21:51, Adam Kocoloski <kocolosk@apache.org> wrote:
>>
>>> Were there any view indexes on that DB?  My suspicion is that
>>> couch_view_group misses a step somewhere in its dance and keeps a reference
>>> to an old #db{}
>>>
>>> Adam
>>>
>>> On Sep 14, 2010, at 4:46 PM, [mRg] wrote:
>>>
>>> > It was just on a DB with a lot of writes (documents having lots of
>>> > revisions, rather that lots of new docs), the database grows as expected
>>> but
>>> > after just doing a simple compact on the db the old files still seem to
>>> be
>>> > on disk as reported previously.
>>> >
>>> > It seems its been around a while now as (as far as we can see) 0.11 /
>>> 1.0
>>> > and 1.0.1 all have the same issue, when we replicated the same data over
>>> to
>>> > the newer versions the same identical problem happened.
>>> >
>>> > On 14 September 2010 14:49, Robert Newson <robert.newson@gmail.com>
>>> wrote:
>>> >
>>> >> This was mentioned at CouchCamp, several developers suspect there's
a
>>> >> subtle bug in the code where file descriptors are not being closed
>>> >> under some conditions.
>>> >>
>>> >> Did the OP identify a particular sequence that led here?
>>> >>
>>> >> B.
>>> >>
>>> >> On Mon, Sep 13, 2010 at 6:25 PM, Juan Jose Comellas <
>>> juanjo@comellas.org>
>>> >> wrote:
>>> >>> I'm pretty certain it's a bug in CouchDB. I haven't checked the
code,
>>> but
>>> >>> what is happening is that CouchDB is keeping file descriptors
>>> >> corresponding
>>> >>> to deleted files open, probably after compacting a database. I'm
>>> >> inferring
>>> >>> this based on the fact that with lsof I saw several cases of two
files
>>> >> with
>>> >>> the same name (one deleted and one not) opened by CouchDB.
>>> >>>
>>> >>>
>>> >>> On Mon, Sep 13, 2010 at 7:28 PM, [mRg] <emargee@gmail.com>
wrote:
>>> >>>
>>> >>>> Is this an issue with the OS or with Couch/Beam itself ?
>>> >>>>
>>> >>>> On 10 September 2010 16:57, Juan Jose Comellas <juanjo@comellas.org>
>>> >>>> wrote:
>>> >>>>
>>> >>>>> It also happened to me with Ubuntu 8.04 and CouchDB 0.10.x.
The only
>>> >> way
>>> >>>> I
>>> >>>>> found to solve the problem was by restarting CouchDB.
>>> >>>>>
>>> >>>>>
>>> >>>>> On Thu, Sep 9, 2010 at 10:25 AM, [mRg] <emargee@gmail.com>
wrote:
>>> >>>>>
>>> >>>>>> [CouchDB Version : 0.11]
>>> >>>>>> [OS: RHEL 5.5]
>>> >>>>>>
>>> >>>>>> Hi all,
>>> >>>>>>
>>> >>>>>> We are seeing some odd behaviour with CouchDb not releasing
files
>>> >> after
>>> >>>>> it
>>> >>>>>> has compacted them meaning we are seeing disk space
continue to
>>> >>>> increase
>>> >>>>>> until the service is restarted even though we have regular
>>> >> compaction /
>>> >>>>>> cleanup.
>>> >>>>>>
>>> >>>>>> Typical scenario : File system alert for disk space
..
>>> >>>>>>
>>> >>>>>> df says 5.5.GB used ..
>>> >>>>>>
>>> >>>>>> df -h | grep -P 'Filesystem|db'
>>> >>>>>>
>>> >>>>>> Filesystem            Size  Used Avail Use% Mounted
on
>>> >>>>>>
>>> >>>>>>                     8.0G  5.5G  2.2G  72%
/var/lib/db
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> .. but the files on the disk (and in Futon) say only
793MB is being
>>> >>>> used
>>> >>>>> ..
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> du -sh /var/lib/db/*
>>> >>>>>>
>>> >>>>>> 793M     /var/lib/db/couchdb
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> .. running lsof shows the files marked as deleted but
still there
>>> >> (I'm
>>> >>>>>> assuming beam still has some kind of handle still open
to them) ..
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> lsof | grep -P 'COMMAND|/var/lib/db/couchdb/'
>>> >>>>>>
>>> >>>>>> COMMAND     PID      USER   FD      TYPE  
          DEVICE
>>> >> SIZE
>>> >>>>>> NODE NAME
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   12u      REG    
        253,10
>>> >> 4183
>>> >>>>>> 1210374
>>> >>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>
>>> /var/lib/db/couchdb/.active_context_design/8efe9584a944f4b24b97f134f541d80f.view
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   16u      REG    
        253,10
>>> >> 612039080
>>> >>>>>> 1204227 /var/lib/db/couchdb/active_context.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   20u      REG    
        253,10
>>> >> 180699242
>>> >>>>>> 1204237 /var/lib/db/couchdb/active_context.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   21u      REG    
        253,10
>>> >> 3133540
>>> >>>>>> 2023429 /var/lib/db/couchdb/thesaurus.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   25u      REG    
        253,10
>>> >> 71422052
>>> >>>>>> 2023428 /var/lib/db/couchdb/active_context_schedule.couch
(deleted)
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   26u      REG    
        253,10
>>> >> 71372900
>>> >>>>>> 2056193 /var/lib/db/couchdb/active_context_schedule.couch
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   28u      REG    
        253,10
>>> >> 621252717
>>> >>>>>> 1204238 /var/lib/db/couchdb/active_context.couch
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   29u      REG    
        253,10
>>> >> 1740903
>>> >>>>>> 90113 /var/lib/db/couchdb/mapping_cache.couch
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   30u      REG    
        253,10
>>> >> 3133540
>>> >>>>>> 1204253 /var/lib/db/couchdb/thesaurus.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   31u      REG    
        253,10
>>> >> 2318439
>>> >>>>>> 1204240 /var/lib/db/couchdb/mappings.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   32u      REG    
        253,10
>>> >> 983150
>>> >>>>>> 2004994
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>
>>> /var/lib/db/couchdb/.mappings_design/f9964c50b95c167c8ac3d8d3b4c76a4b.view
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   33u      REG    
        253,10
>>> >> 3133540
>>> >>>>>> 2056194 /var/lib/db/couchdb/thesaurus.couch
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   34u      REG    
        253,10
>>> >> 70742116
>>> >>>>>> 1204231 /var/lib/db/couchdb/active_context_schedule.couch
(deleted)
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   44u      REG    
        253,10
>>> >> 1740903
>>> >>>>>> 2023427 /var/lib/db/couchdb/mapping_cache.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   45u      REG    
        253,10
>>> >> 65396836
>>> >>>>>> 1994756 /var/lib/db/couchdb/active_context_schedule.couch
(deleted)
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   47u      REG    
        253,10
>>> >> 71229540
>>> >>>>>> 1204243 /var/lib/db/couchdb/active_context_schedule.couch
(deleted)
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   48u      REG    
        253,10
>>> >> 240254078
>>> >>>>>>  1204226 /var/lib/db/couchdb/active_context.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   49u      REG    
        253,10
>>> >> 2232423
>>> >>>>>> 1204241 /var/lib/db/couchdb/mappings.couch
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   51u      REG    
        253,10
>>> >> 96903353
>>> >>>>>> 1210371
>>> >>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>
>>> /var/lib/db/couchdb/.active_context_design/b406f7bb6d01b6d50324f14cde588324.view
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   52u      REG    
        253,10
>>> >> 1192034
>>> >>>>>> 1249282
>>> >>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>
>>> /var/lib/db/couchdb/.mapping_cache_design/8bbf80a6e1ca8f2c71ebc90c730e7190.view
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   53u      REG    
        253,10
>>> >> 1622135
>>> >>>>>> 2004993
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>
>>> /var/lib/db/couchdb/.mappings_design/1ff69004b575837aae9d4147162197d4.view
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   54u      REG    
        253,10
>>> >> 4172
>>> >>>>>> 1208327
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>
>>> /var/lib/db/couchdb/.thesaurus_design/73fe79b20ff06e341257fa78231ca87e.view
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   55u      REG    
        253,10
>>> >> 13496483
>>> >>>>>> 1208325
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>
>>> /var/lib/db/couchdb/.thesaurus_design/fcc08d6e41c54c0d470c5984ea9eb6e3.view
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   56u      REG    
        253,10
>>> >> 12959902
>>> >>>>>> 1208326
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>
>>> /var/lib/db/couchdb/.thesaurus_design/c06195da19f34ced37fa1ab91e663c73.view
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   57u      REG    
        253,10
>>> >> 70254692
>>> >>>>>> 1204236 /var/lib/db/couchdb/active_context_schedule.couch
(deleted)
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   58u      REG    
        253,10
>>> >> 581730
>>> >>>>>> 1210373
>>> >>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>
>>> /var/lib/db/couchdb/.active_context_schedule_design/c64c047dcff74ea2122356efb1042759.view
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   67u      REG    
        253,10
>>> >> 1818727
>>> >>>>>> 1204247 /var/lib/db/couchdb/mapping_cache.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   78u      REG    
        253,10
>>> >> 609905146
>>> >>>>>> 1204234 /var/lib/db/couchdb/active_context.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   80u      REG    
        253,10
>>> >> 545919085
>>> >>>>>> 1994753 /var/lib/db/couchdb/active_context.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   81u      REG    
        253,10
>>> >> 71032932
>>> >>>>>> 1204249 /var/lib/db/couchdb/active_context_schedule.couch
(deleted)
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   84u      REG    
        253,10
>>> >> 548866111
>>> >>>>>> 1204239 /var/lib/db/couchdb/active_context.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   85u      REG    
        253,10
>>> >> 564612402
>>> >>>>>> 1204233 /var/lib/db/couchdb/active_context.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb   87u      REG    
        253,10
>>> >> 615996179
>>> >>>>>> 1204228 /var/lib/db/couchdb/active_context.couch (deleted)
>>> >>>>>>
>>> >>>>>> beam.smp  21352   couchdb  102u      REG    
        253,10
>>> >> 561111149
>>> >>>>>> 2023425 /var/lib/db/couchdb/active_context.couch (deleted)
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> Has anyone seen this behaviour before or knows of a
way around this
>>> >> ?
>>> >>>>>> Currently we have to restart the couchdb service in
order to
>>> restore
>>> >>>> the
>>> >>>>>> disk space.
>>> >>>>>>
>>> >>>>>> Any help/advice is much appreciated.
>>> >>>>>>
>>> >>>>>> Regards
>>> >>>>>>
>>> >>>>>> Stephen
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>>
>>> >>
>>>
>>>
>>
>

Mime
View raw message