incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Filipe David Manana <fdman...@apache.org>
Subject Re: Couch not releasing deleted files
Date Mon, 22 Nov 2010 12:00:09 GMT
Last week I was able to reproduce and fixed it.
https://issues.apache.org/jira/browse/COUCHDB-926 is now closed.

The same type of issue happened with continuous replications and the
the authentication cache (_users database).
Both fixed in trunk, 1.1.x and 1.0.x.

1.0.2 and 1.1.0 will contain the fixes.

On Tue, Sep 28, 2010 at 1:27 AM, Panop S. <pas@vizrt.com> wrote:
> Hi,
>
>   Thanks, how to apply this ?
>   By replacing this couchdb/trunk/src/couchdb/couch_file.erl file or I
> have to build it all again ?
>
> Bests,
> Aun.
>
> On Mon, 2010-09-27 at 12:14 +0200, [mRg] wrote:
>> Anyone still following this on here, I asked this question on the dev list
>> and Filipe has committed a potential fix for this issue here:
>>
>> http://svn.apache.org/viewvc?view=revision&revision=1001196
>>
>> <http://svn.apache.org/viewvc?view=revision&revision=1001196>if people
are
>> still having this issue they could try applying this, we are now doing
>> internal testing to see if this has fixed it for our use case.
>>
>> Regards
>>
>> Stephen
>>
>> On 19 September 2010 00:36, Robert Newson <robert.newson@gmail.com> wrote:
>>
>> > It's been discussed by dev's at CouchCamp but a mailing list reminder
>> > couldn't hurt. There's some discussion about 1.1, so perhaps that's a
>> > good thread to raise this on.
>> >
>> > B.
>> >
>> > On Sun, Sep 19, 2010 at 12:24 AM, [mRg] <emargee@gmail.com> wrote:
>> > > Has this been raised with the dev list or is there a bug raised for this
>> > > issue ? Seems a few people are having this and more are probably not
>> > seeing
>> > > yet.
>> > >
>> > > On 15 September 2010 08:49, [mRg] <emargee@gmail.com> wrote:
>> > >
>> > >> Yes there are 6 views (2 design docs)
>> > >>
>> > >>
>> > >> On 14 September 2010 21:51, Adam Kocoloski <kocolosk@apache.org>
wrote:
>> > >>
>> > >>> Were there any view indexes on that DB?  My suspicion is that
>> > >>> couch_view_group misses a step somewhere in its dance and keeps
a
>> > reference
>> > >>> to an old #db{}
>> > >>>
>> > >>> Adam
>> > >>>
>> > >>> On Sep 14, 2010, at 4:46 PM, [mRg] wrote:
>> > >>>
>> > >>> > It was just on a DB with a lot of writes (documents having
lots of
>> > >>> > revisions, rather that lots of new docs), the database grows
as
>> > expected
>> > >>> but
>> > >>> > after just doing a simple compact on the db the old files
still seem
>> > to
>> > >>> be
>> > >>> > on disk as reported previously.
>> > >>> >
>> > >>> > It seems its been around a while now as (as far as we can
see) 0.11 /
>> > >>> 1.0
>> > >>> > and 1.0.1 all have the same issue, when we replicated the
same data
>> > over
>> > >>> to
>> > >>> > the newer versions the same identical problem happened.
>> > >>> >
>> > >>> > On 14 September 2010 14:49, Robert Newson <robert.newson@gmail.com>
>> > >>> wrote:
>> > >>> >
>> > >>> >> This was mentioned at CouchCamp, several developers suspect
there's
>> > a
>> > >>> >> subtle bug in the code where file descriptors are not
being closed
>> > >>> >> under some conditions.
>> > >>> >>
>> > >>> >> Did the OP identify a particular sequence that led here?
>> > >>> >>
>> > >>> >> B.
>> > >>> >>
>> > >>> >> On Mon, Sep 13, 2010 at 6:25 PM, Juan Jose Comellas <
>> > >>> juanjo@comellas.org>
>> > >>> >> wrote:
>> > >>> >>> I'm pretty certain it's a bug in CouchDB. I haven't
checked the
>> > code,
>> > >>> but
>> > >>> >>> what is happening is that CouchDB is keeping file
descriptors
>> > >>> >> corresponding
>> > >>> >>> to deleted files open, probably after compacting a
database. I'm
>> > >>> >> inferring
>> > >>> >>> this based on the fact that with lsof I saw several
cases of two
>> > files
>> > >>> >> with
>> > >>> >>> the same name (one deleted and one not) opened by
CouchDB.
>> > >>> >>>
>> > >>> >>>
>> > >>> >>> On Mon, Sep 13, 2010 at 7:28 PM, [mRg] <emargee@gmail.com>
wrote:
>> > >>> >>>
>> > >>> >>>> Is this an issue with the OS or with Couch/Beam
itself ?
>> > >>> >>>>
>> > >>> >>>> On 10 September 2010 16:57, Juan Jose Comellas
<
>> > juanjo@comellas.org>
>> > >>> >>>> wrote:
>> > >>> >>>>
>> > >>> >>>>> It also happened to me with Ubuntu 8.04 and
CouchDB 0.10.x. The
>> > only
>> > >>> >> way
>> > >>> >>>> I
>> > >>> >>>>> found to solve the problem was by restarting
CouchDB.
>> > >>> >>>>>
>> > >>> >>>>>
>> > >>> >>>>> On Thu, Sep 9, 2010 at 10:25 AM, [mRg] <emargee@gmail.com>
>> > wrote:
>> > >>> >>>>>
>> > >>> >>>>>> [CouchDB Version : 0.11]
>> > >>> >>>>>> [OS: RHEL 5.5]
>> > >>> >>>>>>
>> > >>> >>>>>> Hi all,
>> > >>> >>>>>>
>> > >>> >>>>>> We are seeing some odd behaviour with
CouchDb not releasing
>> > files
>> > >>> >> after
>> > >>> >>>>> it
>> > >>> >>>>>> has compacted them meaning we are seeing
disk space continue to
>> > >>> >>>> increase
>> > >>> >>>>>> until the service is restarted even though
we have regular
>> > >>> >> compaction /
>> > >>> >>>>>> cleanup.
>> > >>> >>>>>>
>> > >>> >>>>>> Typical scenario : File system alert for
disk space ..
>> > >>> >>>>>>
>> > >>> >>>>>> df says 5.5.GB used ..
>> > >>> >>>>>>
>> > >>> >>>>>> df -h | grep -P 'Filesystem|db'
>> > >>> >>>>>>
>> > >>> >>>>>> Filesystem            Size  Used
Avail Use% Mounted on
>> > >>> >>>>>>
>> > >>> >>>>>>                     8.0G  5.5G
 2.2G  72% /var/lib/db
>> > >>> >>>>>>
>> > >>> >>>>>>
>> > >>> >>>>>> .. but the files on the disk (and in Futon)
say only 793MB is
>> > being
>> > >>> >>>> used
>> > >>> >>>>> ..
>> > >>> >>>>>>
>> > >>> >>>>>>
>> > >>> >>>>>> du -sh /var/lib/db/*
>> > >>> >>>>>>
>> > >>> >>>>>> 793M     /var/lib/db/couchdb
>> > >>> >>>>>>
>> > >>> >>>>>>
>> > >>> >>>>>> .. running lsof shows the files marked
as deleted but still
>> > there
>> > >>> >> (I'm
>> > >>> >>>>>> assuming beam still has some kind of handle
still open to them)
>> > ..
>> > >>> >>>>>>
>> > >>> >>>>>>
>> > >>> >>>>>> lsof | grep -P 'COMMAND|/var/lib/db/couchdb/'
>> > >>> >>>>>>
>> > >>> >>>>>> COMMAND     PID      USER   FD  
   TYPE             DEVICE
>> > >>> >> SIZE
>> > >>> >>>>>> NODE NAME
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   12u  
   REG             253,10
>> > >>> >> 4183
>> > >>> >>>>>> 1210374
>> > >>> >>>>>>
>> > >>> >>>>>>
>> > >>> >>>>>
>> > >>> >>>>
>> > >>> >>
>> > >>>
>> > /var/lib/db/couchdb/.active_context_design/8efe9584a944f4b24b97f134f541d80f.view
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   16u  
   REG             253,10
>> > >>> >> 612039080
>> > >>> >>>>>> 1204227 /var/lib/db/couchdb/active_context.couch
(deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   20u  
   REG             253,10
>> > >>> >> 180699242
>> > >>> >>>>>> 1204237 /var/lib/db/couchdb/active_context.couch
(deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   21u  
   REG             253,10
>> > >>> >> 3133540
>> > >>> >>>>>> 2023429 /var/lib/db/couchdb/thesaurus.couch
(deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   25u  
   REG             253,10
>> > >>> >> 71422052
>> > >>> >>>>>> 2023428 /var/lib/db/couchdb/active_context_schedule.couch
>> > (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   26u  
   REG             253,10
>> > >>> >> 71372900
>> > >>> >>>>>> 2056193 /var/lib/db/couchdb/active_context_schedule.couch
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   28u  
   REG             253,10
>> > >>> >> 621252717
>> > >>> >>>>>> 1204238 /var/lib/db/couchdb/active_context.couch
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   29u  
   REG             253,10
>> > >>> >> 1740903
>> > >>> >>>>>> 90113 /var/lib/db/couchdb/mapping_cache.couch
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   30u  
   REG             253,10
>> > >>> >> 3133540
>> > >>> >>>>>> 1204253 /var/lib/db/couchdb/thesaurus.couch
(deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   31u  
   REG             253,10
>> > >>> >> 2318439
>> > >>> >>>>>> 1204240 /var/lib/db/couchdb/mappings.couch
(deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   32u  
   REG             253,10
>> > >>> >> 983150
>> > >>> >>>>>> 2004994
>> > >>> >>>>>>
>> > >>> >>>>>
>> > >>> >>>>
>> > >>> >>
>> > >>>
>> > /var/lib/db/couchdb/.mappings_design/f9964c50b95c167c8ac3d8d3b4c76a4b.view
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   33u  
   REG             253,10
>> > >>> >> 3133540
>> > >>> >>>>>> 2056194 /var/lib/db/couchdb/thesaurus.couch
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   34u  
   REG             253,10
>> > >>> >> 70742116
>> > >>> >>>>>> 1204231 /var/lib/db/couchdb/active_context_schedule.couch
>> > (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   44u  
   REG             253,10
>> > >>> >> 1740903
>> > >>> >>>>>> 2023427 /var/lib/db/couchdb/mapping_cache.couch
(deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   45u  
   REG             253,10
>> > >>> >> 65396836
>> > >>> >>>>>> 1994756 /var/lib/db/couchdb/active_context_schedule.couch
>> > (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   47u  
   REG             253,10
>> > >>> >> 71229540
>> > >>> >>>>>> 1204243 /var/lib/db/couchdb/active_context_schedule.couch
>> > (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   48u  
   REG             253,10
>> > >>> >> 240254078
>> > >>> >>>>>>  1204226 /var/lib/db/couchdb/active_context.couch
(deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   49u  
   REG             253,10
>> > >>> >> 2232423
>> > >>> >>>>>> 1204241 /var/lib/db/couchdb/mappings.couch
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   51u  
   REG             253,10
>> > >>> >> 96903353
>> > >>> >>>>>> 1210371
>> > >>> >>>>>>
>> > >>> >>>>>>
>> > >>> >>>>>
>> > >>> >>>>
>> > >>> >>
>> > >>>
>> > /var/lib/db/couchdb/.active_context_design/b406f7bb6d01b6d50324f14cde588324.view
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   52u  
   REG             253,10
>> > >>> >> 1192034
>> > >>> >>>>>> 1249282
>> > >>> >>>>>>
>> > >>> >>>>>>
>> > >>> >>>>>
>> > >>> >>>>
>> > >>> >>
>> > >>>
>> > /var/lib/db/couchdb/.mapping_cache_design/8bbf80a6e1ca8f2c71ebc90c730e7190.view
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   53u  
   REG             253,10
>> > >>> >> 1622135
>> > >>> >>>>>> 2004993
>> > >>> >>>>>>
>> > >>> >>>>>
>> > >>> >>>>
>> > >>> >>
>> > >>>
>> > /var/lib/db/couchdb/.mappings_design/1ff69004b575837aae9d4147162197d4.view
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   54u  
   REG             253,10
>> > >>> >> 4172
>> > >>> >>>>>> 1208327
>> > >>> >>>>>>
>> > >>> >>>>>
>> > >>> >>>>
>> > >>> >>
>> > >>>
>> > /var/lib/db/couchdb/.thesaurus_design/73fe79b20ff06e341257fa78231ca87e.view
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   55u  
   REG             253,10
>> > >>> >> 13496483
>> > >>> >>>>>> 1208325
>> > >>> >>>>>>
>> > >>> >>>>>
>> > >>> >>>>
>> > >>> >>
>> > >>>
>> > /var/lib/db/couchdb/.thesaurus_design/fcc08d6e41c54c0d470c5984ea9eb6e3.view
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   56u  
   REG             253,10
>> > >>> >> 12959902
>> > >>> >>>>>> 1208326
>> > >>> >>>>>>
>> > >>> >>>>>
>> > >>> >>>>
>> > >>> >>
>> > >>>
>> > /var/lib/db/couchdb/.thesaurus_design/c06195da19f34ced37fa1ab91e663c73.view
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   57u  
   REG             253,10
>> > >>> >> 70254692
>> > >>> >>>>>> 1204236 /var/lib/db/couchdb/active_context_schedule.couch
>> > (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   58u  
   REG             253,10
>> > >>> >> 581730
>> > >>> >>>>>> 1210373
>> > >>> >>>>>>
>> > >>> >>>>>>
>> > >>> >>>>>
>> > >>> >>>>
>> > >>> >>
>> > >>>
>> > /var/lib/db/couchdb/.active_context_schedule_design/c64c047dcff74ea2122356efb1042759.view
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   67u  
   REG             253,10
>> > >>> >> 1818727
>> > >>> >>>>>> 1204247 /var/lib/db/couchdb/mapping_cache.couch
(deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   78u  
   REG             253,10
>> > >>> >> 609905146
>> > >>> >>>>>> 1204234 /var/lib/db/couchdb/active_context.couch
(deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   80u  
   REG             253,10
>> > >>> >> 545919085
>> > >>> >>>>>> 1994753 /var/lib/db/couchdb/active_context.couch
(deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   81u  
   REG             253,10
>> > >>> >> 71032932
>> > >>> >>>>>> 1204249 /var/lib/db/couchdb/active_context_schedule.couch
>> > (deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   84u  
   REG             253,10
>> > >>> >> 548866111
>> > >>> >>>>>> 1204239 /var/lib/db/couchdb/active_context.couch
(deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   85u  
   REG             253,10
>> > >>> >> 564612402
>> > >>> >>>>>> 1204233 /var/lib/db/couchdb/active_context.couch
(deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb   87u  
   REG             253,10
>> > >>> >> 615996179
>> > >>> >>>>>> 1204228 /var/lib/db/couchdb/active_context.couch
(deleted)
>> > >>> >>>>>>
>> > >>> >>>>>> beam.smp  21352   couchdb  102u  
   REG             253,10
>> > >>> >> 561111149
>> > >>> >>>>>> 2023425 /var/lib/db/couchdb/active_context.couch
(deleted)
>> > >>> >>>>>>
>> > >>> >>>>>>
>> > >>> >>>>>> Has anyone seen this behaviour before
or knows of a way around
>> > this
>> > >>> >> ?
>> > >>> >>>>>> Currently we have to restart the couchdb
service in order to
>> > >>> restore
>> > >>> >>>> the
>> > >>> >>>>>> disk space.
>> > >>> >>>>>>
>> > >>> >>>>>> Any help/advice is much appreciated.
>> > >>> >>>>>>
>> > >>> >>>>>> Regards
>> > >>> >>>>>>
>> > >>> >>>>>> Stephen
>> > >>> >>>>>>
>> > >>> >>>>>
>> > >>> >>>>
>> > >>> >>>
>> > >>> >>
>> > >>>
>> > >>>
>> > >>
>> > >
>> >
>
>
>



-- 
Filipe David Manana,
fdmanana@gmail.com, fdmanana@apache.org

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

Mime
View raw message