incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Eisenmann <si...@struktur.de>
Subject Re: Compact not completing
Date Thu, 11 Aug 2011 16:08:17 GMT
Hi Adam,

i wrote a short python script, which loads the complete changes feed and
requests all of the documents not marked "deleted" using HTTP HEAD
requests. Though i only find documents which have been deleted after i
have retrieved the complete changes view (takes a while for large
databases), and never the same.

So again no luck here in finding the problems.

Any more suggestions? Is there a way to rebuild the complete database
file (maybe offline?).

Thank you and best regards
Simon


ps: The script i have been using is attached.



Am Dienstag, den 09.08.2011, 16:10 -0400 schrieb Adam Kocoloski:
> Hi Simon, CouchDB 1.1.0 includes a recent optimization to _changes?include_docs=true
which allows it to skip a lookup in the id tree and instead load the document body from the
pointer in the sequence tree.  In that case you wouldn't notice any missing entry in the id
tree.  You would notice it, however, if you did direct lookups for each document. Apologies
for the outdated instructions.  Can you try looking up the documents in a separate request
and see if the results change?
> 
> Adam
> 
> On Aug 9, 2011, at 5:49 AM, Simon Eisenmann wrote:
> 
> > Hi Adam,
> > 
> > i just checked the whole _changes feed (since=0) and could not find any
> > document "missing" when using "include_docs=true".
> > 
> > The database in itself has only around 30 documents, so should be quite
> > small. Though there are lots of creations and deletions happening all
> > the time. Thus it its daily purged and compacted. 
> > 
> > So - the changes feed is of no help. Any other idea?
> > 
> > Thank you and best regards
> > Simon
> > 
> > Am Montag, den 08.08.2011, 13:25 -0400 schrieb Adam Kocoloski:
> >> Hi Simon, I think my amended instructions to Mike are still a sensible way to
debug/workaround the problem.  Reiterating (96282148 was the last seq Mike observed in the
Futon status for the compaction):
> >> 
> >>> 1) What you really want are the last 1000 Ids in the seq_tree prior to the
compactor crash.  So maybe something like
> >>> 
> >>> GET /iris/_changes?descending=true&limit=1000&since=96282148
> >> 
> >>> 2) Figure out which of those entries are missing from the id tree, e.g.
lookup the document and see if the response is {"not_found":"missing"}.  You could also try
using include_docs=true on the _changes feed to accomplish the same.
> >> 
> >>> 3) Once you've identified the problematic IDs, try creating them again.
 You might end up introducing duplicates in the _changes feed, but if you do there's a procedure
to fix that.
> >> 
> >> Regards, Adam
> >> 
> >> On Aug 8, 2011, at 12:31 PM, Simon Eisenmann wrote:
> >> 
> >>> Hi Guys,
> >>> 
> >>> i have a couple of CouchDB instances which started to be come
> >>> unpackable. It shows this error:
> >>> 
> >>> [Mon, 08 Aug 2011 16:16:27 GMT] [info] [<0.10808.123>] Starting
> >>> compaction for db "database1"
> >>> [Mon, 08 Aug 2011 16:16:45 GMT] [error] [<0.10808.123>] ** Generic
> >>> server <0.10808.123> terminating 
> >>> ** Last message in was {'EXIT',<0.30396.143>,
> >>>                       {function_clause,
> >>>                        [{couch_db_updater,'-copy_docs/4-fun-2-',
> >>>                          [not_found,
> >>>                           {db,<0.10807.123>,<0.10808.123>,nil,
> >>>                            <<"1312767347007568">>,<0.10805.123>,
> >>>                            <0.10809.123>,
> >>>                            {db_header,5,6340261,0,
> >>>                             {7198006895,{65952,10145}},
> >>>                             {7198010315,76813},
> >>>                             {7198051016,[]},
> >>>                             364,7050915618,nil,1000},
> >>>                            6340261,
> >>>                            {btree,<0.10805.123>,
> >>>                             {7198006895,{65952,10145}},
> >>>                             #Fun<couch_db_updater.10.19222179>,
> >>>                             #Fun<couch_db_updater.11.21515767>,
> >>>                             #Fun<couch_btree.5.124754102>,
> >>>                             #Fun<couch_db_updater.12.93888648>},
> >>>                            {btree,<0.10805.123>,
> >>>                             {7198010315,76813},
> >>>                             #Fun<couch_db_updater.13.40165027>,
> >>>                             #Fun<couch_db_updater.14.82810239>,
> >>>                             #Fun<couch_btree.5.124754102>,
> >>>                             #Fun<couch_db_updater.15.104121193>},
> >>>                            {btree,<0.10805.123>,
> >>>                             {7198051016,[]},
> >>>                             #Fun<couch_btree.0.83553141>,
> >>>                             #Fun<couch_btree.1.30790806>,
> >>>                             #Fun<couch_btree.2.124754102>,nil},
> >>>                            6340261,<<"spreedcom_accounts_1">>,
> >>>                            "/var/lib/couchdb/database1.couch",[],
> >>>                            [],nil,
> >>>                            {user_ctx,null,[],undefined},
> >>>                            nil,1000,
> >>>                            [before_header,after_header,on_file_open],
> >>>                            false},
> >>>                           <0.30397.143>]},
> >>>                         {lists,map,2},
> >>>                         {lists,map,2},
> >>>                         {couch_db_updater,copy_docs,4},
> >>>                         {couch_db_updater,'-copy_compact/3-fun-0-',6},
> >>>                         {couch_btree,stream_kv_node2,8},
> >>>                         {couch_btree,stream_kp_node,7},
> >>>                         {couch_btree,fold,4}]}}
> >>> 
> >>> 
> >>> ... lots of more similar errors following.
> >>> 
> >>> In this mailing list i have found a similar issue from the beginning of
> >>> this year (Fri, 31 Dec 2010 12:38:18), though without a solution.
> >>> 
> >>> This database got purged a lot and has constant changes. So pretty
> >>> similar from the older topic. So i assume there is something wrong in
> >>> the database file related to previous purges. So looks like some bug
> >>> here.
> >>> 
> >>> So - any hints how to fix this? The database is getting pretty large and
> >>> has to be packed from time to time.
> >>> 
> >>> CouchDB is running with Version 1.1.0 on Linux 64bit. The database has
> >>> initially been created with CouchDB 1.0.1 - though the issue appeared a
> >>> couple of weeks ago (packing has been working with 1.1.0 before.
> >>> 
> >>> 
> >>> Thank you and best regards
> >>> Simon
> >>> 
> >>> 
> >>> -- 
> >>> Simon Eisenmann
> >>> 
> >>> [ mailto:simon@struktur.de ]
> >>> 
> >>> [ struktur AG | Kronenstraße 22a | D-70173 Stuttgart ]
> >>> [ T. +49.711.896656.0 | F.+49.711.89665610 ]
> >>> [ http://www.struktur.de | mailto:info@struktur.de ]
> >> 
> > 
> > -- 
> > Simon Eisenmann
> > 
> > [ mailto:simon@struktur.de ]
> > 
> > [ struktur AG | Kronenstraße 22a | D-70173 Stuttgart ]
> > [ T. +49.711.896656.68 | F.+49.711.89665610 ]
> > [ http://www.struktur.de | mailto:info@struktur.de ]
> 

-- 
Simon Eisenmann

[ mailto:simon@struktur.de ]

[ struktur AG | Kronenstraße 22a | D-70173 Stuttgart ]
[ T. +49.711.896656.68 | F.+49.711.89665610 ]
[ http://www.struktur.de | mailto:info@struktur.de ]

Mime
View raw message