lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless" <luc...@mikemccandless.com>
Subject Re: restoring a corrupt index?
Date Sat, 10 Nov 2007 22:01:19 GMT

"Yonik Seeley" <yonik@apache.org> wrote:

> > How can this lead to index corruption?  The "no such file or directory" on
> > loading _cf9.fnm sounds like index corruption?
> 
> I don't think older versions of lucene handled these errors as well.
> Perhaps _cf9.fnm failed to be written, but the segments file succeeded.
> It could also be Solr's fault for allowing further operations on an
> index after one failed?  I'm not sure how that should be handled.

Ahh, OK.

Yeah it's not clear how Solr should handle this case, though Lucene
should be at (get to?) the point where on exception no harm to the
index can be done.  Ie, index on disk is left consistent, and, the
state of the writer is such that it can't corrupt the index even if
it's further used after the exception.

> > Or maybe the index is not corrupt but then we are hitting the descriptor limit
> > on opening a searcher and it's being reported as "no such file or directory"?
> 
> Hmmm, yes that's possible too...
> Should be easy to tell by checking if the file _cf9.fnm already exists.

Oh yeah.  Ryan, does that file exist?

> > Yonik do you understand why so many unreferenced files are being produced
> > here?  What's the root cause?
> 
> Just guesses... but perhaps new index files are written but things
> fail before the stage where old files are deleted?

OK, though, writer doesn't use many descriptors at all, right?  It
opens 1 segment's worth to flush, and mergeFactor+1 segment's worth to
merge.  It's spooky if Lucene can create zillions of un-referenced
files.  Would be good to get to the root cause & make sure it's really
fixed on trunk.

Or, maybe they are all referenced -- I think there were issues with
the old merge policy that could cause segments to not be merged when
they should have been?

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message