incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Eisenmann <si...@struktur.de>
Subject Re: Compact not completing
Date Fri, 12 Aug 2011 15:57:47 GMT
Woot! The script works and i was able to free around 100GB of space. If
perfectly finds the broken docs, recreates them empty and deletes them
again. Afterwards the databases become compactable again.

Thanks alot Adam for your support.

Attached is the final version of the repair script, which might be of
help for anybody which has this problem

Best regards
Simon

Am Freitag, den 12.08.2011, 10:56 -0400 schrieb Adam Kocoloski:
> Hi Simon, progress!  Yes, go ahead and re-create and delete documents with those IDs
and see if you can successfully compact.  Regards,
> 
> Adam
> 
> On Aug 12, 2011, at 6:00 AM, Simon Eisenmann wrote:
> 
> > Hi Adam,
> > 
> > all right, looks like i am able to find the trouble makers when also
> > looking at the deleted documents:
> > 
> > ~# python repaircompact.py http://localhost:5984/gangstercluster_1
> > Changes feed is
> > http://localhost:5984/gangstercluster_1/_changes?since=0.
> > Fetched feed into /tmp/tmpLjRkZY.
> > Not found error: missing!=deleted
> > (/gangstercluster_1/alive_1dcc4efdb2a411e0acd3003048679e10)
> > Not found error: missing!=deleted
> > (/gangstercluster_1/alive_1dcc523bb2a411e0acd4003048679e10)
> > Not found error: missing!=deleted
> > (/gangstercluster_1/alive_41ed63c1b2a411e0acd4003048679e10)
> > Not found error: missing!=deleted
> > (/gangstercluster_1/alive_41ed678bb2a411e0acd5003048679e10)
> > Processed 8225 entries (last sequence: 5972567)
> > Document count: 8222 (count: 74, deleted: 8148)
> > Document commited sequence is now: 5972569
> > 
> > 
> > Though its hard to say if the number of changes feed rows matches the
> > number of the database infos as its constantly changing. Though its very
> > close so probably it would match if nothing changes.
> > 
> > So now that i have found the missing documents. Should i just create an
> > empty document with that ID, and delete it again?
> > 
> > Thank you and best regards
> > Simon
> > 
> > ps: updated script attached
> > 
> > 
> > Am Donnerstag, den 11.08.2011, 12:30 -0400 schrieb Adam Kocoloski:
> >> Hi Simon, I wouldn't skip the deleted documents, a deleted doc could just as
easily be the one missing in the ID index.  When you lookup a deleted document you should
see "reason":"deleted" instead of the "reason":"missing" that you get if the ID is not in
the index at all.  Then again, if you see that only ever see deleted docs after retrieving
the full _changes then a deleted doc is probably not the source of your troubles.
> >> 
> >> Can you confirm that the number of rows in the changes feed is equal to doc_count
+ doc_del_count from db.info()?  Best,
> >> 
> >> Adam
> >> 
> >> On Aug 11, 2011, at 12:08 PM, Simon Eisenmann wrote:
> >> 
> >>> Hi Adam,
> >>> 
> >>> i wrote a short python script, which loads the complete changes feed and
> >>> requests all of the documents not marked "deleted" using HTTP HEAD
> >>> requests. Though i only find documents which have been deleted after i
> >>> have retrieved the complete changes view (takes a while for large
> >>> databases), and never the same.
> >>> 
> >>> So again no luck here in finding the problems.
> >>> 
> >>> Any more suggestions? Is there a way to rebuild the complete database
> >>> file (maybe offline?).
> >>> 
> >>> Thank you and best regards
> >>> Simon
> >>> 
> >>> 
> >>> ps: The script i have been using is attached.
> >>> 
> >>> 
> >>> 
> >>> Am Dienstag, den 09.08.2011, 16:10 -0400 schrieb Adam Kocoloski:
> >>>> Hi Simon, CouchDB 1.1.0 includes a recent optimization to _changes?include_docs=true
which allows it to skip a lookup in the id tree and instead load the document body from the
pointer in the sequence tree.  In that case you wouldn't notice any missing entry in the id
tree.  You would notice it, however, if you did direct lookups for each document. Apologies
for the outdated instructions.  Can you try looking up the documents in a separate request
and see if the results change?
> >>>> 
> >>>> Adam
> >>>> 
> >>>> On Aug 9, 2011, at 5:49 AM, Simon Eisenmann wrote:
> >>>> 
> >>>>> Hi Adam,
> >>>>> 
> >>>>> i just checked the whole _changes feed (since=0) and could not find
any
> >>>>> document "missing" when using "include_docs=true".
> >>>>> 
> >>>>> The database in itself has only around 30 documents, so should be
quite
> >>>>> small. Though there are lots of creations and deletions happening
all
> >>>>> the time. Thus it its daily purged and compacted. 
> >>>>> 
> >>>>> So - the changes feed is of no help. Any other idea?
> >>>>> 
> >>>>> Thank you and best regards
> >>>>> Simon
> >>>>> 
> >>>>> Am Montag, den 08.08.2011, 13:25 -0400 schrieb Adam Kocoloski:
> >>>>>> Hi Simon, I think my amended instructions to Mike are still
a sensible way to debug/workaround the problem.  Reiterating (96282148 was the last seq Mike
observed in the Futon status for the compaction):
> >>>>>> 
> >>>>>>> 1) What you really want are the last 1000 Ids in the seq_tree
prior to the compactor crash.  So maybe something like
> >>>>>>> 
> >>>>>>> GET /iris/_changes?descending=true&limit=1000&since=96282148
> >>>>>> 
> >>>>>>> 2) Figure out which of those entries are missing from the
id tree, e.g. lookup the document and see if the response is {"not_found":"missing"}.  You
could also try using include_docs=true on the _changes feed to accomplish the same.
> >>>>>> 
> >>>>>>> 3) Once you've identified the problematic IDs, try creating
them again.  You might end up introducing duplicates in the _changes feed, but if you do there's
a procedure to fix that.
> >>>>>> 
> >>>>>> Regards, Adam
> >>>>>> 
> >>>>>> On Aug 8, 2011, at 12:31 PM, Simon Eisenmann wrote:
> >>>>>> 
> >>>>>>> Hi Guys,
> >>>>>>> 
> >>>>>>> i have a couple of CouchDB instances which started to be
come
> >>>>>>> unpackable. It shows this error:
> >>>>>>> 
> >>>>>>> [Mon, 08 Aug 2011 16:16:27 GMT] [info] [<0.10808.123>]
Starting
> >>>>>>> compaction for db "database1"
> >>>>>>> [Mon, 08 Aug 2011 16:16:45 GMT] [error] [<0.10808.123>]
** Generic
> >>>>>>> server <0.10808.123> terminating 
> >>>>>>> ** Last message in was {'EXIT',<0.30396.143>,
> >>>>>>>                     {function_clause,
> >>>>>>>                      [{couch_db_updater,'-copy_docs/4-fun-2-',
> >>>>>>>                        [not_found,
> >>>>>>>                         {db,<0.10807.123>,<0.10808.123>,nil,
> >>>>>>>                          <<"1312767347007568">>,<0.10805.123>,
> >>>>>>>                          <0.10809.123>,
> >>>>>>>                          {db_header,5,6340261,0,
> >>>>>>>                           {7198006895,{65952,10145}},
> >>>>>>>                           {7198010315,76813},
> >>>>>>>                           {7198051016,[]},
> >>>>>>>                           364,7050915618,nil,1000},
> >>>>>>>                          6340261,
> >>>>>>>                          {btree,<0.10805.123>,
> >>>>>>>                           {7198006895,{65952,10145}},
> >>>>>>>                           #Fun<couch_db_updater.10.19222179>,
> >>>>>>>                           #Fun<couch_db_updater.11.21515767>,
> >>>>>>>                           #Fun<couch_btree.5.124754102>,
> >>>>>>>                           #Fun<couch_db_updater.12.93888648>},
> >>>>>>>                          {btree,<0.10805.123>,
> >>>>>>>                           {7198010315,76813},
> >>>>>>>                           #Fun<couch_db_updater.13.40165027>,
> >>>>>>>                           #Fun<couch_db_updater.14.82810239>,
> >>>>>>>                           #Fun<couch_btree.5.124754102>,
> >>>>>>>                           #Fun<couch_db_updater.15.104121193>},
> >>>>>>>                          {btree,<0.10805.123>,
> >>>>>>>                           {7198051016,[]},
> >>>>>>>                           #Fun<couch_btree.0.83553141>,
> >>>>>>>                           #Fun<couch_btree.1.30790806>,
> >>>>>>>                           #Fun<couch_btree.2.124754102>,nil},
> >>>>>>>                          6340261,<<"spreedcom_accounts_1">>,
> >>>>>>>                          "/var/lib/couchdb/database1.couch",[],
> >>>>>>>                          [],nil,
> >>>>>>>                          {user_ctx,null,[],undefined},
> >>>>>>>                          nil,1000,
> >>>>>>>                          [before_header,after_header,on_file_open],
> >>>>>>>                          false},
> >>>>>>>                         <0.30397.143>]},
> >>>>>>>                       {lists,map,2},
> >>>>>>>                       {lists,map,2},
> >>>>>>>                       {couch_db_updater,copy_docs,4},
> >>>>>>>                       {couch_db_updater,'-copy_compact/3-fun-0-',6},
> >>>>>>>                       {couch_btree,stream_kv_node2,8},
> >>>>>>>                       {couch_btree,stream_kp_node,7},
> >>>>>>>                       {couch_btree,fold,4}]}}
> >>>>>>> 
> >>>>>>> 
> >>>>>>> ... lots of more similar errors following.
> >>>>>>> 
> >>>>>>> In this mailing list i have found a similar issue from the
beginning of
> >>>>>>> this year (Fri, 31 Dec 2010 12:38:18), though without a
solution.
> >>>>>>> 
> >>>>>>> This database got purged a lot and has constant changes.
So pretty
> >>>>>>> similar from the older topic. So i assume there is something
wrong in
> >>>>>>> the database file related to previous purges. So looks like
some bug
> >>>>>>> here.
> >>>>>>> 
> >>>>>>> So - any hints how to fix this? The database is getting
pretty large and
> >>>>>>> has to be packed from time to time.
> >>>>>>> 
> >>>>>>> CouchDB is running with Version 1.1.0 on Linux 64bit. The
database has
> >>>>>>> initially been created with CouchDB 1.0.1 - though the issue
appeared a
> >>>>>>> couple of weeks ago (packing has been working with 1.1.0
before.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Thank you and best regards
> >>>>>>> Simon
> >>>>>>> 
> >>>>>>> 
> >>>>>>> -- 
> >>>>>>> Simon Eisenmann
> >>>>>>> 
> >>>>>>> [ mailto:simon@struktur.de ]
> >>>>>>> 
> >>>>>>> [ struktur AG | Kronenstraße 22a | D-70173 Stuttgart ]
> >>>>>>> [ T. +49.711.896656.0 | F.+49.711.89665610 ]
> >>>>>>> [ http://www.struktur.de | mailto:info@struktur.de ]
> >>>>>> 
> >>>>> 
> >>>>> -- 
> >>>>> Simon Eisenmann
> >>>>> 
> >>>>> [ mailto:simon@struktur.de ]
> >>>>> 
> >>>>> [ struktur AG | Kronenstraße 22a | D-70173 Stuttgart ]
> >>>>> [ T. +49.711.896656.68 | F.+49.711.89665610 ]
> >>>>> [ http://www.struktur.de | mailto:info@struktur.de ]
> >>>> 
> >>> 
> >>> -- 
> >>> Simon Eisenmann
> >>> 
> >>> [ mailto:simon@struktur.de ]
> >>> 
> >>> [ struktur AG | Kronenstraße 22a | D-70173 Stuttgart ]
> >>> [ T. +49.711.896656.68 | F.+49.711.89665610 ]
> >>> [ http://www.struktur.de | mailto:info@struktur.de ]
> >>> <repaircompact.py>
> >> 
> > 
> > -- 
> > Simon Eisenmann
> > 
> > [ mailto:simon@struktur.de ]
> > 
> > [ struktur AG | Kronenstraße 22a | D-70173 Stuttgart ]
> > [ T. +49.711.896656.68 | F.+49.711.89665610 ]
> > [ http://www.struktur.de | mailto:info@struktur.de ]
> > <repaircompact.py>
> 

-- 
Simon Eisenmann

[ mailto:simon@struktur.de ]

[ struktur AG | Kronenstraße 22a | D-70173 Stuttgart ]
[ T. +49.711.896656.68 | F.+49.711.89665610 ]
[ http://www.struktur.de | mailto:info@struktur.de ]

Mime
View raw message