lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: detected corrupted index / performance improvement
Date Thu, 07 Feb 2008 21:06:57 GMT

robert engels wrote:

> I might be misunderstanding 1044.  There were several approaches,  
> and I am not certain what was the final???

The final approach (take 7) is to make the index consistent (sync the  
files) after finishing a merge.  Also, a new method ("commit") is  
added which will force a synchronous sync while you wait.  Close also  
does this.

> I reread the bug and am still a bit unclear.
>
> If the segments are sync'd as part of the commit, then yes, that  
> would suffice. The merges don't need to commit, you just can't  
> delete the segments until the merge completes.
>
> I  think that building the segments, and syncing each segment -  
> since in most cases the caller is going to call commit as part of  
> each update, is going to be slower than writing the documents/ 
> operations to a log file, but a lot depends on how Lucene is used  
> (interactive vs. batch, lots of updates vs. a few).

Well, and based on how frequently you prune the transaction log (sync  
the real files).  I think the 2X IO cost is going to make performance  
worse with the transaction log.

> I am not sure how deletions are impacted by all of this.

Should be fine?  The *.del files need to be sync'd just like the rest  
of the segments files.

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message