couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Glenn Rempe <gl...@rempe.us>
Subject Re: Timeout Error when trying to access views + Indexing problems
Date Tue, 06 Oct 2009 15:53:13 GMT
Would replicating the DB to the same host perform those checks?  Also, if I
setup the auto-index every X # of records script shown on the wiki would
that be run on indexing?  These two combined might allow me to essentially
scan and check the records as migrated, and build indexes incrementally from
the get go.
Is there another way to run a scan for 'invalid' records across the whole
db?  Could I write a script to loop through all records?  And if I did what
would I be looking for?  That the JSON parses?  What else?

Also, a new data point.  Last night before going to bed I split up my 7
views which were all in one design doc into 4 docs (1 x 1view, 3 x 2views).
 I started indexing one of them last night and this morning its still
running.  Its a simple view with a map and reduce:

function(doc) {
  if( (doc['couchrest-type'] == 'SearchDocument') && doc.engine) {
    emit(doc.engine, 1);
  }
}

function(keys, values, rereduce) {
  return sum(values);
}

Processed 6570762 of 28249510 changes (23%)

6 million records is higher than I gotten on previous attempts which seemed
to bork at around ~4mm.

Strange.

G

On Tue, Oct 6, 2009 at 6:29 AM, Curt Arnold <carnold@apache.org> wrote:

>
> On Oct 6, 2009, at 1:46 AM, Glenn Rempe wrote:
>
>>
>> - there is some kind of corruption in the main DB file and when this point
>> is reached (or a specific record in the DB?) that it crashes? If so how
>> can
>> I best identify this?
>>
>
> Inserting mal-encoded documents into CouchDB could interfere with document
> retrieval and indexing, see
> https://issues.apache.org/jira/browse/COUCHDB-345.  Possibly one of those
> got into your database and now it is stopping the rebuilding of views.  A
> patch recently got added to prevent mal-encoded documents from being
> accepted, but it does not fix the problem on an existing database that has
> been corrupted.   I do not know if the symptoms are the same as what you are
> observing, but I think it would be a likely culprit.
>



-- 
Glenn Rempe

email                 : glenn@rempe.us
voice                 : (415) 894-5366 or (415)-89G-LENN
twitter                : @grempe
contact info        : http://www.rempe.us/contact.html
pgp                    : http://www.rempe.us/gnupg.txt

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message