couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Eisenmann <si...@struktur.de>
Subject Re: Compact not completing
Date Fri, 12 Aug 2011 10:00:48 GMT
Hi Adam,

all right, looks like i am able to find the trouble makers when also
looking at the deleted documents:

~# python repaircompact.py http://localhost:5984/gangstercluster_1
Changes feed is
http://localhost:5984/gangstercluster_1/_changes?since=0.
Fetched feed into /tmp/tmpLjRkZY.
Not found error: missing!=deleted
(/gangstercluster_1/alive_1dcc4efdb2a411e0acd3003048679e10)
Not found error: missing!=deleted
(/gangstercluster_1/alive_1dcc523bb2a411e0acd4003048679e10)
Not found error: missing!=deleted
(/gangstercluster_1/alive_41ed63c1b2a411e0acd4003048679e10)
Not found error: missing!=deleted
(/gangstercluster_1/alive_41ed678bb2a411e0acd5003048679e10)
Processed 8225 entries (last sequence: 5972567)
Document count: 8222 (count: 74, deleted: 8148)
Document commited sequence is now: 5972569


Though its hard to say if the number of changes feed rows matches the
number of the database infos as its constantly changing. Though its very
close so probably it would match if nothing changes.

So now that i have found the missing documents. Should i just create an
empty document with that ID, and delete it again?

Thank you and best regards
Simon

ps: updated script attached


Am Donnerstag, den 11.08.2011, 12:30 -0400 schrieb Adam Kocoloski:
> Hi Simon, I wouldn't skip the deleted documents, a deleted doc could just as easily be
the one missing in the ID index.  When you lookup a deleted document you should see "reason":"deleted"
instead of the "reason":"missing" that you get if the ID is not in the index at all.  Then
again, if you see that only ever see deleted docs after retrieving the full _changes then
a deleted doc is probably not the source of your troubles.
> 
> Can you confirm that the number of rows in the changes feed is equal to doc_count + doc_del_count
from db.info()?  Best,
> 
> Adam
> 
> On Aug 11, 2011, at 12:08 PM, Simon Eisenmann wrote:
> 
> > Hi Adam,
> > 
> > i wrote a short python script, which loads the complete changes feed and
> > requests all of the documents not marked "deleted" using HTTP HEAD
> > requests. Though i only find documents which have been deleted after i
> > have retrieved the complete changes view (takes a while for large
> > databases), and never the same.
> > 
> > So again no luck here in finding the problems.
> > 
> > Any more suggestions? Is there a way to rebuild the complete database
> > file (maybe offline?).
> > 
> > Thank you and best regards
> > Simon
> > 
> > 
> > ps: The script i have been using is attached.
> > 
> > 
> > 
> > Am Dienstag, den 09.08.2011, 16:10 -0400 schrieb Adam Kocoloski:
> >> Hi Simon, CouchDB 1.1.0 includes a recent optimization to _changes?include_docs=true
which allows it to skip a lookup in the id tree and instead load the document body from the
pointer in the sequence tree.  In that case you wouldn't notice any missing entry in the id
tree.  You would notice it, however, if you did direct lookups for each document. Apologies
for the outdated instructions.  Can you try looking up the documents in a separate request
and see if the results change?
> >> 
> >> Adam
> >> 
> >> On Aug 9, 2011, at 5:49 AM, Simon Eisenmann wrote:
> >> 
> >>> Hi Adam,
> >>> 
> >>> i just checked the whole _changes feed (since=0) and could not find any
> >>> document "missing" when using "include_docs=true".
> >>> 
> >>> The database in itself has only around 30 documents, so should be quite
> >>> small. Though there are lots of creations and deletions happening all
> >>> the time. Thus it its daily purged and compacted. 
> >>> 
> >>> So - the changes feed is of no help. Any other idea?
> >>> 
> >>> Thank you and best regards
> >>> Simon
> >>> 
> >>> Am Montag, den 08.08.2011, 13:25 -0400 schrieb Adam Kocoloski:
> >>>> Hi Simon, I think my amended instructions to Mike are still a sensible
way to debug/workaround the problem.  Reiterating (96282148 was the last seq Mike observed
in the Futon status for the compaction):
> >>>> 
> >>>>> 1) What you really want are the last 1000 Ids in the seq_tree prior
to the compactor crash.  So maybe something like
> >>>>> 
> >>>>> GET /iris/_changes?descending=true&limit=1000&since=96282148
> >>>> 
> >>>>> 2) Figure out which of those entries are missing from the id tree,
e.g. lookup the document and see if the response is {"not_found":"missing"}.  You could also
try using include_docs=true on the _changes feed to accomplish the same.
> >>>> 
> >>>>> 3) Once you've identified the problematic IDs, try creating them
again.  You might end up introducing duplicates in the _changes feed, but if you do there's
a procedure to fix that.
> >>>> 
> >>>> Regards, Adam
> >>>> 
> >>>> On Aug 8, 2011, at 12:31 PM, Simon Eisenmann wrote:
> >>>> 
> >>>>> Hi Guys,
> >>>>> 
> >>>>> i have a couple of CouchDB instances which started to be come
> >>>>> unpackable. It shows this error:
> >>>>> 
> >>>>> [Mon, 08 Aug 2011 16:16:27 GMT] [info] [<0.10808.123>] Starting
> >>>>> compaction for db "database1"
> >>>>> [Mon, 08 Aug 2011 16:16:45 GMT] [error] [<0.10808.123>] **
Generic
> >>>>> server <0.10808.123> terminating 
> >>>>> ** Last message in was {'EXIT',<0.30396.143>,
> >>>>>                      {function_clause,
> >>>>>                       [{couch_db_updater,'-copy_docs/4-fun-2-',
> >>>>>                         [not_found,
> >>>>>                          {db,<0.10807.123>,<0.10808.123>,nil,
> >>>>>                           <<"1312767347007568">>,<0.10805.123>,
> >>>>>                           <0.10809.123>,
> >>>>>                           {db_header,5,6340261,0,
> >>>>>                            {7198006895,{65952,10145}},
> >>>>>                            {7198010315,76813},
> >>>>>                            {7198051016,[]},
> >>>>>                            364,7050915618,nil,1000},
> >>>>>                           6340261,
> >>>>>                           {btree,<0.10805.123>,
> >>>>>                            {7198006895,{65952,10145}},
> >>>>>                            #Fun<couch_db_updater.10.19222179>,
> >>>>>                            #Fun<couch_db_updater.11.21515767>,
> >>>>>                            #Fun<couch_btree.5.124754102>,
> >>>>>                            #Fun<couch_db_updater.12.93888648>},
> >>>>>                           {btree,<0.10805.123>,
> >>>>>                            {7198010315,76813},
> >>>>>                            #Fun<couch_db_updater.13.40165027>,
> >>>>>                            #Fun<couch_db_updater.14.82810239>,
> >>>>>                            #Fun<couch_btree.5.124754102>,
> >>>>>                            #Fun<couch_db_updater.15.104121193>},
> >>>>>                           {btree,<0.10805.123>,
> >>>>>                            {7198051016,[]},
> >>>>>                            #Fun<couch_btree.0.83553141>,
> >>>>>                            #Fun<couch_btree.1.30790806>,
> >>>>>                            #Fun<couch_btree.2.124754102>,nil},
> >>>>>                           6340261,<<"spreedcom_accounts_1">>,
> >>>>>                           "/var/lib/couchdb/database1.couch",[],
> >>>>>                           [],nil,
> >>>>>                           {user_ctx,null,[],undefined},
> >>>>>                           nil,1000,
> >>>>>                           [before_header,after_header,on_file_open],
> >>>>>                           false},
> >>>>>                          <0.30397.143>]},
> >>>>>                        {lists,map,2},
> >>>>>                        {lists,map,2},
> >>>>>                        {couch_db_updater,copy_docs,4},
> >>>>>                        {couch_db_updater,'-copy_compact/3-fun-0-',6},
> >>>>>                        {couch_btree,stream_kv_node2,8},
> >>>>>                        {couch_btree,stream_kp_node,7},
> >>>>>                        {couch_btree,fold,4}]}}
> >>>>> 
> >>>>> 
> >>>>> ... lots of more similar errors following.
> >>>>> 
> >>>>> In this mailing list i have found a similar issue from the beginning
of
> >>>>> this year (Fri, 31 Dec 2010 12:38:18), though without a solution.
> >>>>> 
> >>>>> This database got purged a lot and has constant changes. So pretty
> >>>>> similar from the older topic. So i assume there is something wrong
in
> >>>>> the database file related to previous purges. So looks like some
bug
> >>>>> here.
> >>>>> 
> >>>>> So - any hints how to fix this? The database is getting pretty large
and
> >>>>> has to be packed from time to time.
> >>>>> 
> >>>>> CouchDB is running with Version 1.1.0 on Linux 64bit. The database
has
> >>>>> initially been created with CouchDB 1.0.1 - though the issue appeared
a
> >>>>> couple of weeks ago (packing has been working with 1.1.0 before.
> >>>>> 
> >>>>> 
> >>>>> Thank you and best regards
> >>>>> Simon
> >>>>> 
> >>>>> 
> >>>>> -- 
> >>>>> Simon Eisenmann
> >>>>> 
> >>>>> [ mailto:simon@struktur.de ]
> >>>>> 
> >>>>> [ struktur AG | Kronenstraße 22a | D-70173 Stuttgart ]
> >>>>> [ T. +49.711.896656.0 | F.+49.711.89665610 ]
> >>>>> [ http://www.struktur.de | mailto:info@struktur.de ]
> >>>> 
> >>> 
> >>> -- 
> >>> Simon Eisenmann
> >>> 
> >>> [ mailto:simon@struktur.de ]
> >>> 
> >>> [ struktur AG | Kronenstraße 22a | D-70173 Stuttgart ]
> >>> [ T. +49.711.896656.68 | F.+49.711.89665610 ]
> >>> [ http://www.struktur.de | mailto:info@struktur.de ]
> >> 
> > 
> > -- 
> > Simon Eisenmann
> > 
> > [ mailto:simon@struktur.de ]
> > 
> > [ struktur AG | Kronenstraße 22a | D-70173 Stuttgart ]
> > [ T. +49.711.896656.68 | F.+49.711.89665610 ]
> > [ http://www.struktur.de | mailto:info@struktur.de ]
> > <repaircompact.py>
> 

-- 
Simon Eisenmann

[ mailto:simon@struktur.de ]

[ struktur AG | Kronenstraße 22a | D-70173 Stuttgart ]
[ T. +49.711.896656.68 | F.+49.711.89665610 ]
[ http://www.struktur.de | mailto:info@struktur.de ]

Mime
View raw message