Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of paul.joseph.davis@gmail.com
 designates 209.85.161.180 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=ID3sV/4vxACg4ymx/7OGUKyj2nla/N8bY7grfSupyuerZsYZB/SoBincnzPRPWg9bO
         3lVXZAJVZf3uuqOH6QnuikXj/OUIh/wroLYAyg/d2UxiUAc38Zjl1LKFzUr1UqTn8QcA
         7koRXHoEXu9EHm6kmI30HEwkY58BDIoJKphcg=
MIME-Version: 1.0
In-Reply-To: <AANLkTi=6vCEqS5enHQx=-U_T0koWNxfaVkg3d9_tbaxi@mail.gmail.com>
References: <AANLkTinmRGyCjVoKEceV5w6eYJdPDAW9WJY9MvRvWM57@mail.gmail.com>
 <AANLkTimbFKGnbX-WtFn7B=TG7Gkhq1evfEiuRD4e6=U-@mail.gmail.com>
 <AANLkTimKfUG273G8etu40O6wOAwbzsY3rvPN77Ju8B0P@mail.gmail.com>
 <AANLkTikzt32LCjHuKSH0CS18h4U9ZXg=apyBJ+=oZYFL@mail.gmail.com>
 <AANLkTi=6vCEqS5enHQx=-U_T0koWNxfaVkg3d9_tbaxi@mail.gmail.com>
From: Paul Davis <paul.joseph.davis@gmail.com>
Date: Mon, 20 Sep 2010 16:42:45 -0400
Message-ID: <AANLkTinchBr=REHg2t=ce2w4nhvb1JvogHJgSV1rTfO3@mail.gmail.com>
Subject: Re: CouchDb not releasing files
To: dev@couchdb.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Do you issue two compact commands simultaneously or wait for the first
one to complete and then run the second?


On Mon, Sep 20, 2010 at 4:07 PM, [mRg] <emargee@gmail.com> wrote:
> Well I can certainly reproduce the issue, but am having trouble finding t=
he
> exact sequence (annoyingly of course)
>
> 1> I started with a blank VM of Ubuntu 10.10 (ext3) running on virtualbox
> with latest CouchDB (apt-get install couchdb)
> 2> Began adding lots of blank docs .. =A0(curl -X POST -H
> "Content-Type:application/json" http://localhost:5984/test -d {})
> 3> Created a simple view ("function(doc) { =A0emit(null, doc); }") and ra=
n to
> ensure it wrote to disk.
> 4> I issued 2 compact commands (curl -X POST
> http://localhost:5984/test/_compact/ -H "Content-Type:application/json")
> 5> And began adding more documents again as before ..
>
> Repeating this, essentially creating view data and compacting it about 10
> times resulted in 4 'snagged' files
>
> root@ubuntu:~# lsof | grep -P 'COMMAND|/var/lib/couchdb/1.0.1'
> COMMAND =A0 =A0PID =A0 =A0USER =A0 FD =A0 =A0 =A0TYPE =A0 =A0 DEVICE SIZE=
/OFF =A0 =A0 =A0 NODE NAME
> beam =A0 =A0 =A01083 couchdb =A0 13u =A0 =A0 =A0REG =A0 =A0 =A0252,0 =A0 =
=A0 4185 =A0 =A0 =A0 =A0271
> /var/lib/couchdb/1.0.1/_users.couch
>>> beam =A0 =A0 =A01083 couchdb =A0 14u =A0 =A0 =A0REG =A0 =A0 =A0252,0 =
=A0 331865 =A0 =A0 =A0 =A0275
> /var/lib/couchdb/1.0.1/.delete/4e000e0ae5cce3d275f09b4542396e85 (deleted)
>>> beam =A0 =A0 =A01083 couchdb =A0 16u =A0 =A0 =A0REG =A0 =A0 =A0252,0 =
=A0 602210 =A0 =A0 =A0 =A0279
> /var/lib/couchdb/1.0.1/.delete/a9ee2b96c12591455ea795fa5324dc9c (deleted)
> beam =A0 =A0 =A01083 couchdb =A0 17u =A0 =A0 =A0REG =A0 =A0 =A0252,0 =A0 =
270434 =A0 =A0 =A0 =A0283
> /var/lib/couchdb/1.0.1/.test_design/07ca32cf9b0de9c915c5d9ce653cdca3.view
> beam =A0 =A0 =A01083 couchdb =A0 18u =A0 =A0 =A0REG =A0 =A0 =A0252,0 =A0 =
204898 =A0 =A0 =A0 =A0284
> /var/lib/couchdb/1.0.1/.test_design/540958c4124af3925fe467afb96f4906.view
> beam =A0 =A0 =A01083 couchdb =A0 20u =A0 =A0 =A0REG =A0 =A0 =A0252,0 =A0 =
405602 =A0 =A0 =A0 =A0286
> /var/lib/couchdb/1.0.1/.test_design/f26a8fcc3d2ce226a9e652338882c3db.view
> beam =A0 =A0 =A01083 couchdb =A0 21u =A0 =A0 =A0REG =A0 =A0 =A0252,0 =A0 =
299106 =A0 =A0 =A0 =A0287
> /var/lib/couchdb/1.0.1/.test_design/40614f8c8e1b4bab9d093881e914729d.view
>>> beam =A0 =A0 =A01083 couchdb =A0 22u =A0 =A0 =A0REG =A0 =A0 =A0252,0 =
=A0 106594 =A0 =A0 =A0 =A0285
> /var/lib/couchdb/1.0.1/.delete/31909e936ce94db7a6cede72827b18b2 (deleted)
>>> beam =A0 =A0 =A01083 couchdb =A0 24r =A0 =A0 =A0REG =A0 =A0 =A0252,0 =
=A0 233570 =A0 =A0 =A0 =A0288
> /var/lib/couchdb/1.0.1/.delete/83d09ec6dab90ca6078c0310085b97cc (deleted)
> beam =A0 =A0 =A01083 couchdb =A0 26u =A0 =A0 =A0REG =A0 =A0 =A0252,0 =A0 =
163932 =A0 =A0 =A0 =A0281
> /var/lib/couchdb/1.0.1/.test_design/3486e8de398e27b8767afd4975691360.view
> beam =A0 =A0 =A01083 couchdb =A0 27w =A0 =A0 =A0REG =A0 =A0 =A0252,0 =A0 =
114786 =A0 =A0 =A0 =A0289
> /var/lib/couchdb/1.0.1/test.couch
> beam =A0 =A0 =A01083 couchdb =A0 28u =A0 =A0 =A0REG =A0 =A0 =A0252,0 =A0 =
217186 =A0 =A0 =A0 =A0280
> /var/lib/couchdb/1.0.1/.test_design/b450740e3245f89ab902db10d767a397.view
>
> .. while the bottom 2 marked with (deleted) hung around for about 20 minu=
tes
> they eventually disappeared ..
>
> root@ubuntu:~# lsof | grep -P 'COMMAND|/var/lib/couchdb/1.0.1'
> COMMAND =A0 =A0PID =A0 =A0USER =A0 FD =A0 =A0 =A0TYPE =A0 =A0 DEVICE SIZE=
/OFF =A0 =A0 =A0 NODE NAME
> beam =A0 =A0 =A01083 couchdb =A0 13u =A0 =A0 =A0REG =A0 =A0 =A0252,0 =A0 =
=A0 4185 =A0 =A0 =A0 =A0271
> /var/lib/couchdb/1.0.1/_users.couch
>>> beam =A0 =A0 =A01083 couchdb =A0 14u =A0 =A0 =A0REG =A0 =A0 =A0252,0 =
=A0 331865 =A0 =A0 =A0 =A0275
> /var/lib/couchdb/1.0.1/.delete/4e000e0ae5cce3d275f09b4542396e85 (deleted)
>>> beam =A0 =A0 =A01083 couchdb =A0 16u =A0 =A0 =A0REG =A0 =A0 =A0252,0 =
=A0 602210 =A0 =A0 =A0 =A0279
> /var/lib/couchdb/1.0.1/.delete/a9ee2b96c12591455ea795fa5324dc9c (deleted)
> beam =A0 =A0 =A01083 couchdb =A0 17u =A0 =A0 =A0REG =A0 =A0 =A0252,0 =A0 =
270434 =A0 =A0 =A0 =A0283
> /var/lib/couchdb/1.0.1/.test_design/07ca32cf9b0de9c915c5d9ce653cdca3.view
> beam =A0 =A0 =A01083 couchdb =A0 18u =A0 =A0 =A0REG =A0 =A0 =A0252,0 =A0 =
204898 =A0 =A0 =A0 =A0284
> /var/lib/couchdb/1.0.1/.test_design/540958c4124af3925fe467afb96f4906.view
> beam =A0 =A0 =A01083 couchdb =A0 20u =A0 =A0 =A0REG =A0 =A0 =A0252,0 =A0 =
405602 =A0 =A0 =A0 =A0286
> /var/lib/couchdb/1.0.1/.test_design/f26a8fcc3d2ce226a9e652338882c3db.view
> beam =A0 =A0 =A01083 couchdb =A0 21u =A0 =A0 =A0REG =A0 =A0 =A0252,0 =A0 =
299106 =A0 =A0 =A0 =A0287
> /var/lib/couchdb/1.0.1/.test_design/40614f8c8e1b4bab9d093881e914729d.view
> beam =A0 =A0 =A01083 couchdb =A0 23u =A0 =A0 =A0REG =A0 =A0 =A0252,0 =A0 =
114786 =A0 =A0 =A0 =A0288
> /var/lib/couchdb/1.0.1/test.couch
> beam =A0 =A0 =A01083 couchdb =A0 26u =A0 =A0 =A0REG =A0 =A0 =A0252,0 =A0 =
163932 =A0 =A0 =A0 =A0281
> /var/lib/couchdb/1.0.1/.test_design/3486e8de398e27b8767afd4975691360.view
> beam =A0 =A0 =A01083 couchdb =A0 28u =A0 =A0 =A0REG =A0 =A0 =A0252,0 =A0 =
217186 =A0 =A0 =A0 =A0280
> /var/lib/couchdb/1.0.1/.test_design/b450740e3245f89ab902db10d767a397.view
>
> .. but the top 2 did not and are still hanging around over an hour later =
..
>
> I will keep looking to see if I can find the exact sequence ..
>
> On 20 September 2010 19:09, [mRg] <emargee@gmail.com> wrote:
>
>> I am trying to reproduce it currently on a fresh VM. It seems to be a
>> symptom of view compaction, after the file is moved to the .delete direc=
tory
>> and then finally deleted is where the problem seems to occur.
>>
>> I tried with a fresh ubuntu 10.10 but I couldnt reproduce but the system
>> I'm seeing it on is ext3 (ubuntu was ext4) so I'm going to start again w=
ith
>> that and report back.
>>
>> I did wonder how this would be reproduced in a test as you have to get
>> quite low level (using lsof) to see if the issue is manifesting itself :=
)
>>
>> Regards
>>
>> Stephen
>>
>> On 20 September 2010 16:00, Chris Anderson <jchris@apache.org> wrote:
>>
>>> On Sun, Sep 19, 2010 at 3:21 PM, Randall Leeds <randall.leeds@gmail.com=
>
>>> wrote:
>>> > If the bug is confirmed it should be on JIRA if it is not already. If
>>> > you have a test case that reproduces it that would be fanstastic
>>> > (bonus points for a JS test in Futon).
>>> >
>>> > It's my opinion something this serious should block 1.1, but
>>> > ultimately that is up to the committers to determine, yes?
>>> >
>>>
>>> I think this is pretty serious. It's probably a simple fix, so as soon
>>> as we have it reproducible, I'm sure it will get fixed.
>>>
>>> I'm not sure how it can be tested from JavaScript, any ideas?
>>>
>>> If not it should be testable via the Erlang tests.
>>>
>>> Chris
>>>
>>> > On Sun, Sep 19, 2010 at 22:09, [mRg] <emargee@gmail.com> wrote:
>>> >> Hi all,
>>> >>
>>> >> This is just a cross-post to highlight a thread on the user list. (w=
ith
>>> the
>>> >> same name as this one - all details etc are on there, happy to repea=
t
>>> here
>>> >> if needed). It seems this problem was discussed by some devs at
>>> CouchCamp
>>> >> and seems others are suffering this issue. I was just wondering if
>>> there was
>>> >> a JIRA issue related to this that I/we can track and if a fix for th=
is
>>> will
>>> >> be included in any upcoming released (1.1 ?).
>>> >>
>>> >> Regards
>>> >>
>>> >> Stephen
>>> >>
>>> >
>>>
>>>
>>>
>>> --
>>> Chris Anderson
>>> http://jchrisa.net
>>> http://couch.io
>>>
>>
>>
>