incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Kocoloski <kocol...@apache.org>
Subject Re: Compact not completing
Date Fri, 12 Aug 2011 14:56:57 GMT
Hi Simon, progress!  Yes, go ahead and re-create and delete documents with those IDs and see
if you can successfully compact.  Regards,

Adam

On Aug 12, 2011, at 6:00 AM, Simon Eisenmann wrote:

> Hi Adam,
> 
> all right, looks like i am able to find the trouble makers when also
> looking at the deleted documents:
> 
> ~# python repaircompact.py http://localhost:5984/gangstercluster_1
> Changes feed is
> http://localhost:5984/gangstercluster_1/_changes?since=0.
> Fetched feed into /tmp/tmpLjRkZY.
> Not found error: missing!=deleted
> (/gangstercluster_1/alive_1dcc4efdb2a411e0acd3003048679e10)
> Not found error: missing!=deleted
> (/gangstercluster_1/alive_1dcc523bb2a411e0acd4003048679e10)
> Not found error: missing!=deleted
> (/gangstercluster_1/alive_41ed63c1b2a411e0acd4003048679e10)
> Not found error: missing!=deleted
> (/gangstercluster_1/alive_41ed678bb2a411e0acd5003048679e10)
> Processed 8225 entries (last sequence: 5972567)
> Document count: 8222 (count: 74, deleted: 8148)
> Document commited sequence is now: 5972569
> 
> 
> Though its hard to say if the number of changes feed rows matches the
> number of the database infos as its constantly changing. Though its very
> close so probably it would match if nothing changes.
> 
> So now that i have found the missing documents. Should i just create an
> empty document with that ID, and delete it again?
> 
> Thank you and best regards
> Simon
> 
> ps: updated script attached
> 
> 
> Am Donnerstag, den 11.08.2011, 12:30 -0400 schrieb Adam Kocoloski:
>> Hi Simon, I wouldn't skip the deleted documents, a deleted doc could just as easily
be the one missing in the ID index.  When you lookup a deleted document you should see "reason":"deleted"
instead of the "reason":"missing" that you get if the ID is not in the index at all.  Then
again, if you see that only ever see deleted docs after retrieving the full _changes then
a deleted doc is probably not the source of your troubles.
>> 
>> Can you confirm that the number of rows in the changes feed is equal to doc_count
+ doc_del_count from db.info()?  Best,
>> 
>> Adam
>> 
>> On Aug 11, 2011, at 12:08 PM, Simon Eisenmann wrote:
>> 
>>> Hi Adam,
>>> 
>>> i wrote a short python script, which loads the complete changes feed and
>>> requests all of the documents not marked "deleted" using HTTP HEAD
>>> requests. Though i only find documents which have been deleted after i
>>> have retrieved the complete changes view (takes a while for large
>>> databases), and never the same.
>>> 
>>> So again no luck here in finding the problems.
>>> 
>>> Any more suggestions? Is there a way to rebuild the complete database
>>> file (maybe offline?).
>>> 
>>> Thank you and best regards
>>> Simon
>>> 
>>> 
>>> ps: The script i have been using is attached.
>>> 
>>> 
>>> 
>>> Am Dienstag, den 09.08.2011, 16:10 -0400 schrieb Adam Kocoloski:
>>>> Hi Simon, CouchDB 1.1.0 includes a recent optimization to _changes?include_docs=true
which allows it to skip a lookup in the id tree and instead load the document body from the
pointer in the sequence tree.  In that case you wouldn't notice any missing entry in the id
tree.  You would notice it, however, if you did direct lookups for each document. Apologies
for the outdated instructions.  Can you try looking up the documents in a separate request
and see if the results change?
>>>> 
>>>> Adam
>>>> 
>>>> On Aug 9, 2011, at 5:49 AM, Simon Eisenmann wrote:
>>>> 
>>>>> Hi Adam,
>>>>> 
>>>>> i just checked the whole _changes feed (since=0) and could not find any
>>>>> document "missing" when using "include_docs=true".
>>>>> 
>>>>> The database in itself has only around 30 documents, so should be quite
>>>>> small. Though there are lots of creations and deletions happening all
>>>>> the time. Thus it its daily purged and compacted. 
>>>>> 
>>>>> So - the changes feed is of no help. Any other idea?
>>>>> 
>>>>> Thank you and best regards
>>>>> Simon
>>>>> 
>>>>> Am Montag, den 08.08.2011, 13:25 -0400 schrieb Adam Kocoloski:
>>>>>> Hi Simon, I think my amended instructions to Mike are still a sensible
way to debug/workaround the problem.  Reiterating (96282148 was the last seq Mike observed
in the Futon status for the compaction):
>>>>>> 
>>>>>>> 1) What you really want are the last 1000 Ids in the seq_tree
prior to the compactor crash.  So maybe something like
>>>>>>> 
>>>>>>> GET /iris/_changes?descending=true&limit=1000&since=96282148
>>>>>> 
>>>>>>> 2) Figure out which of those entries are missing from the id
tree, e.g. lookup the document and see if the response is {"not_found":"missing"}.  You could
also try using include_docs=true on the _changes feed to accomplish the same.
>>>>>> 
>>>>>>> 3) Once you've identified the problematic IDs, try creating them
again.  You might end up introducing duplicates in the _changes feed, but if you do there's
a procedure to fix that.
>>>>>> 
>>>>>> Regards, Adam
>>>>>> 
>>>>>> On Aug 8, 2011, at 12:31 PM, Simon Eisenmann wrote:
>>>>>> 
>>>>>>> Hi Guys,
>>>>>>> 
>>>>>>> i have a couple of CouchDB instances which started to be come
>>>>>>> unpackable. It shows this error:
>>>>>>> 
>>>>>>> [Mon, 08 Aug 2011 16:16:27 GMT] [info] [<0.10808.123>]
Starting
>>>>>>> compaction for db "database1"
>>>>>>> [Mon, 08 Aug 2011 16:16:45 GMT] [error] [<0.10808.123>]
** Generic
>>>>>>> server <0.10808.123> terminating 
>>>>>>> ** Last message in was {'EXIT',<0.30396.143>,
>>>>>>>                     {function_clause,
>>>>>>>                      [{couch_db_updater,'-copy_docs/4-fun-2-',
>>>>>>>                        [not_found,
>>>>>>>                         {db,<0.10807.123>,<0.10808.123>,nil,
>>>>>>>                          <<"1312767347007568">>,<0.10805.123>,
>>>>>>>                          <0.10809.123>,
>>>>>>>                          {db_header,5,6340261,0,
>>>>>>>                           {7198006895,{65952,10145}},
>>>>>>>                           {7198010315,76813},
>>>>>>>                           {7198051016,[]},
>>>>>>>                           364,7050915618,nil,1000},
>>>>>>>                          6340261,
>>>>>>>                          {btree,<0.10805.123>,
>>>>>>>                           {7198006895,{65952,10145}},
>>>>>>>                           #Fun<couch_db_updater.10.19222179>,
>>>>>>>                           #Fun<couch_db_updater.11.21515767>,
>>>>>>>                           #Fun<couch_btree.5.124754102>,
>>>>>>>                           #Fun<couch_db_updater.12.93888648>},
>>>>>>>                          {btree,<0.10805.123>,
>>>>>>>                           {7198010315,76813},
>>>>>>>                           #Fun<couch_db_updater.13.40165027>,
>>>>>>>                           #Fun<couch_db_updater.14.82810239>,
>>>>>>>                           #Fun<couch_btree.5.124754102>,
>>>>>>>                           #Fun<couch_db_updater.15.104121193>},
>>>>>>>                          {btree,<0.10805.123>,
>>>>>>>                           {7198051016,[]},
>>>>>>>                           #Fun<couch_btree.0.83553141>,
>>>>>>>                           #Fun<couch_btree.1.30790806>,
>>>>>>>                           #Fun<couch_btree.2.124754102>,nil},
>>>>>>>                          6340261,<<"spreedcom_accounts_1">>,
>>>>>>>                          "/var/lib/couchdb/database1.couch",[],
>>>>>>>                          [],nil,
>>>>>>>                          {user_ctx,null,[],undefined},
>>>>>>>                          nil,1000,
>>>>>>>                          [before_header,after_header,on_file_open],
>>>>>>>                          false},
>>>>>>>                         <0.30397.143>]},
>>>>>>>                       {lists,map,2},
>>>>>>>                       {lists,map,2},
>>>>>>>                       {couch_db_updater,copy_docs,4},
>>>>>>>                       {couch_db_updater,'-copy_compact/3-fun-0-',6},
>>>>>>>                       {couch_btree,stream_kv_node2,8},
>>>>>>>                       {couch_btree,stream_kp_node,7},
>>>>>>>                       {couch_btree,fold,4}]}}
>>>>>>> 
>>>>>>> 
>>>>>>> ... lots of more similar errors following.
>>>>>>> 
>>>>>>> In this mailing list i have found a similar issue from the beginning
of
>>>>>>> this year (Fri, 31 Dec 2010 12:38:18), though without a solution.
>>>>>>> 
>>>>>>> This database got purged a lot and has constant changes. So pretty
>>>>>>> similar from the older topic. So i assume there is something
wrong in
>>>>>>> the database file related to previous purges. So looks like some
bug
>>>>>>> here.
>>>>>>> 
>>>>>>> So - any hints how to fix this? The database is getting pretty
large and
>>>>>>> has to be packed from time to time.
>>>>>>> 
>>>>>>> CouchDB is running with Version 1.1.0 on Linux 64bit. The database
has
>>>>>>> initially been created with CouchDB 1.0.1 - though the issue
appeared a
>>>>>>> couple of weeks ago (packing has been working with 1.1.0 before.
>>>>>>> 
>>>>>>> 
>>>>>>> Thank you and best regards
>>>>>>> Simon
>>>>>>> 
>>>>>>> 
>>>>>>> -- 
>>>>>>> Simon Eisenmann
>>>>>>> 
>>>>>>> [ mailto:simon@struktur.de ]
>>>>>>> 
>>>>>>> [ struktur AG | Kronenstraße 22a | D-70173 Stuttgart ]
>>>>>>> [ T. +49.711.896656.0 | F.+49.711.89665610 ]
>>>>>>> [ http://www.struktur.de | mailto:info@struktur.de ]
>>>>>> 
>>>>> 
>>>>> -- 
>>>>> Simon Eisenmann
>>>>> 
>>>>> [ mailto:simon@struktur.de ]
>>>>> 
>>>>> [ struktur AG | Kronenstraße 22a | D-70173 Stuttgart ]
>>>>> [ T. +49.711.896656.68 | F.+49.711.89665610 ]
>>>>> [ http://www.struktur.de | mailto:info@struktur.de ]
>>>> 
>>> 
>>> -- 
>>> Simon Eisenmann
>>> 
>>> [ mailto:simon@struktur.de ]
>>> 
>>> [ struktur AG | Kronenstraße 22a | D-70173 Stuttgart ]
>>> [ T. +49.711.896656.68 | F.+49.711.89665610 ]
>>> [ http://www.struktur.de | mailto:info@struktur.de ]
>>> <repaircompact.py>
>> 
> 
> -- 
> Simon Eisenmann
> 
> [ mailto:simon@struktur.de ]
> 
> [ struktur AG | Kronenstraße 22a | D-70173 Stuttgart ]
> [ T. +49.711.896656.68 | F.+49.711.89665610 ]
> [ http://www.struktur.de | mailto:info@struktur.de ]
> <repaircompact.py>


Mime
View raw message