Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of adam.kocoloski@gmail.com
 designates 209.85.160.180 as permitted sender)
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Compact not completing
From: Adam Kocoloski <kocolosk@apache.org>
In-Reply-To: <1313078897.28552.590.camel@redemption.intranet.struktur.de>
Date: Thu, 11 Aug 2011 12:30:11 -0400
Content-Transfer-Encoding: quoted-printable
Message-Id: <15B4B79E-5C7D-4447-A620-F4D1A663E180@apache.org>
References: <1312821086.28552.374.camel@redemption.intranet.struktur.de>
 <23F087AD-2BB7-444B-A392-DA4DD502960B@apache.org>
 <1312883349.28552.391.camel@redemption.intranet.struktur.de>
 <2084656C-6545-4856-B941-E2D41956D8CE@apache.org>
 <1313078897.28552.590.camel@redemption.intranet.struktur.de>
To: user@couchdb.apache.org

Hi Simon, I wouldn't skip the deleted documents, a deleted doc could =
just as easily be the one missing in the ID index.  When you lookup a =
deleted document you should see "reason":"deleted" instead of the =
"reason":"missing" that you get if the ID is not in the index at all.  =
Then again, if you see that only ever see deleted docs after retrieving =
the full _changes then a deleted doc is probably not the source of your =
troubles.

Can you confirm that the number of rows in the changes feed is equal to =
doc_count + doc_del_count from db.info()?  Best,

Adam

On Aug 11, 2011, at 12:08 PM, Simon Eisenmann wrote:

> Hi Adam,
>=20
> i wrote a short python script, which loads the complete changes feed =
and
> requests all of the documents not marked "deleted" using HTTP HEAD
> requests. Though i only find documents which have been deleted after i
> have retrieved the complete changes view (takes a while for large
> databases), and never the same.
>=20
> So again no luck here in finding the problems.
>=20
> Any more suggestions? Is there a way to rebuild the complete database
> file (maybe offline?).
>=20
> Thank you and best regards
> Simon
>=20
>=20
> ps: The script i have been using is attached.
>=20
>=20
>=20
> Am Dienstag, den 09.08.2011, 16:10 -0400 schrieb Adam Kocoloski:
>> Hi Simon, CouchDB 1.1.0 includes a recent optimization to =
_changes?include_docs=3Dtrue which allows it to skip a lookup in the id =
tree and instead load the document body from the pointer in the sequence =
tree.  In that case you wouldn't notice any missing entry in the id =
tree.  You would notice it, however, if you did direct lookups for each =
document. Apologies for the outdated instructions.  Can you try looking =
up the documents in a separate request and see if the results change?
>>=20
>> Adam
>>=20
>> On Aug 9, 2011, at 5:49 AM, Simon Eisenmann wrote:
>>=20
>>> Hi Adam,
>>>=20
>>> i just checked the whole _changes feed (since=3D0) and could not =
find any
>>> document "missing" when using "include_docs=3Dtrue".
>>>=20
>>> The database in itself has only around 30 documents, so should be =
quite
>>> small. Though there are lots of creations and deletions happening =
all
>>> the time. Thus it its daily purged and compacted.=20
>>>=20
>>> So - the changes feed is of no help. Any other idea?
>>>=20
>>> Thank you and best regards
>>> Simon
>>>=20
>>> Am Montag, den 08.08.2011, 13:25 -0400 schrieb Adam Kocoloski:
>>>> Hi Simon, I think my amended instructions to Mike are still a =
sensible way to debug/workaround the problem.  Reiterating (96282148 was =
the last seq Mike observed in the Futon status for the compaction):
>>>>=20
>>>>> 1) What you really want are the last 1000 Ids in the seq_tree =
prior to the compactor crash.  So maybe something like
>>>>>=20
>>>>> GET /iris/_changes?descending=3Dtrue&limit=3D1000&since=3D96282148
>>>>=20
>>>>> 2) Figure out which of those entries are missing from the id tree, =
e.g. lookup the document and see if the response is =
{"not_found":"missing"}.  You could also try using include_docs=3Dtrue =
on the _changes feed to accomplish the same.
>>>>=20
>>>>> 3) Once you've identified the problematic IDs, try creating them =
again.  You might end up introducing duplicates in the _changes feed, =
but if you do there's a procedure to fix that.
>>>>=20
>>>> Regards, Adam
>>>>=20
>>>> On Aug 8, 2011, at 12:31 PM, Simon Eisenmann wrote:
>>>>=20
>>>>> Hi Guys,
>>>>>=20
>>>>> i have a couple of CouchDB instances which started to be come
>>>>> unpackable. It shows this error:
>>>>>=20
>>>>> [Mon, 08 Aug 2011 16:16:27 GMT] [info] [<0.10808.123>] Starting
>>>>> compaction for db "database1"
>>>>> [Mon, 08 Aug 2011 16:16:45 GMT] [error] [<0.10808.123>] ** Generic
>>>>> server <0.10808.123> terminating=20
>>>>> ** Last message in was {'EXIT',<0.30396.143>,
>>>>>                      {function_clause,
>>>>>                       [{couch_db_updater,'-copy_docs/4-fun-2-',
>>>>>                         [not_found,
>>>>>                          {db,<0.10807.123>,<0.10808.123>,nil,
>>>>>                           <<"1312767347007568">>,<0.10805.123>,
>>>>>                           <0.10809.123>,
>>>>>                           {db_header,5,6340261,0,
>>>>>                            {7198006895,{65952,10145}},
>>>>>                            {7198010315,76813},
>>>>>                            {7198051016,[]},
>>>>>                            364,7050915618,nil,1000},
>>>>>                           6340261,
>>>>>                           {btree,<0.10805.123>,
>>>>>                            {7198006895,{65952,10145}},
>>>>>                            #Fun<couch_db_updater.10.19222179>,
>>>>>                            #Fun<couch_db_updater.11.21515767>,
>>>>>                            #Fun<couch_btree.5.124754102>,
>>>>>                            #Fun<couch_db_updater.12.93888648>},
>>>>>                           {btree,<0.10805.123>,
>>>>>                            {7198010315,76813},
>>>>>                            #Fun<couch_db_updater.13.40165027>,
>>>>>                            #Fun<couch_db_updater.14.82810239>,
>>>>>                            #Fun<couch_btree.5.124754102>,
>>>>>                            #Fun<couch_db_updater.15.104121193>},
>>>>>                           {btree,<0.10805.123>,
>>>>>                            {7198051016,[]},
>>>>>                            #Fun<couch_btree.0.83553141>,
>>>>>                            #Fun<couch_btree.1.30790806>,
>>>>>                            #Fun<couch_btree.2.124754102>,nil},
>>>>>                           6340261,<<"spreedcom_accounts_1">>,
>>>>>                           "/var/lib/couchdb/database1.couch",[],
>>>>>                           [],nil,
>>>>>                           {user_ctx,null,[],undefined},
>>>>>                           nil,1000,
>>>>>                           =
[before_header,after_header,on_file_open],
>>>>>                           false},
>>>>>                          <0.30397.143>]},
>>>>>                        {lists,map,2},
>>>>>                        {lists,map,2},
>>>>>                        {couch_db_updater,copy_docs,4},
>>>>>                        =
{couch_db_updater,'-copy_compact/3-fun-0-',6},
>>>>>                        {couch_btree,stream_kv_node2,8},
>>>>>                        {couch_btree,stream_kp_node,7},
>>>>>                        {couch_btree,fold,4}]}}
>>>>>=20
>>>>>=20
>>>>> ... lots of more similar errors following.
>>>>>=20
>>>>> In this mailing list i have found a similar issue from the =
beginning of
>>>>> this year (Fri, 31 Dec 2010 12:38:18), though without a solution.
>>>>>=20
>>>>> This database got purged a lot and has constant changes. So pretty
>>>>> similar from the older topic. So i assume there is something wrong =
in
>>>>> the database file related to previous purges. So looks like some =
bug
>>>>> here.
>>>>>=20
>>>>> So - any hints how to fix this? The database is getting pretty =
large and
>>>>> has to be packed from time to time.
>>>>>=20
>>>>> CouchDB is running with Version 1.1.0 on Linux 64bit. The database =
has
>>>>> initially been created with CouchDB 1.0.1 - though the issue =
appeared a
>>>>> couple of weeks ago (packing has been working with 1.1.0 before.
>>>>>=20
>>>>>=20
>>>>> Thank you and best regards
>>>>> Simon
>>>>>=20
>>>>>=20
>>>>> --=20
>>>>> Simon Eisenmann
>>>>>=20
>>>>> [ mailto:simon@struktur.de ]
>>>>>=20
>>>>> [ struktur AG | Kronenstra=DFe 22a | D-70173 Stuttgart ]
>>>>> [ T. +49.711.896656.0 | F.+49.711.89665610 ]
>>>>> [ http://www.struktur.de | mailto:info@struktur.de ]
>>>>=20
>>>=20
>>> --=20
>>> Simon Eisenmann
>>>=20
>>> [ mailto:simon@struktur.de ]
>>>=20
>>> [ struktur AG | Kronenstra=DFe 22a | D-70173 Stuttgart ]
>>> [ T. +49.711.896656.68 | F.+49.711.89665610 ]
>>> [ http://www.struktur.de | mailto:info@struktur.de ]
>>=20
>=20
> --=20
> Simon Eisenmann
>=20
> [ mailto:simon@struktur.de ]
>=20
> [ struktur AG | Kronenstra=DFe 22a | D-70173 Stuttgart ]
> [ T. +49.711.896656.68 | F.+49.711.89665610 ]
> [ http://www.struktur.de | mailto:info@struktur.de ]
> <repaircompact.py>