lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Optimization and Corruption Issues
Date Thu, 01 Oct 2009 21:01:21 GMT
On Thu, Oct 1, 2009 at 2:56 PM, Earwin Burrfoot <earwin@gmail.com> wrote:
>>> If you're doing a full optimize, you're still using a single thread. Am I wrong?
>>
>> Depends on how many merges are required, and, the merge scheduler.  In
>> this case (w/ 7000 segments, which is way too many, normally!),
>> assuming ConcurrentMergeScheduler, multiple threads will be used since
>> many merges will be pending.
>>
>> When it gets down to the last (enormous) merge, it's only one thread.
> I'm speaking about full optimize. Is there any way to do it more
> efficiently then running a single, last (enormous) merge?

I guess we could merge different parts of the index w/ different
threads, if we wanted to push concurrency down into a single merge.

> If you try to parallelize, you're merging some documents several times
> (more work) and killing your disks, as merges are mostly IO-bound.

Actually I've found merging of the postings to be CPU bound.  I think
the priority queue, and decode/encode of vInt, are the big CPU costs.

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message