lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Earwin Burrfoot <ear...@gmail.com>
Subject Re: Consolidate MP and LMP
Date Thu, 02 Dec 2010 11:10:29 GMT
> Hmm .. now that I look closely at it, MP has useCompundFile/DocStore
> methods, and LMP just adds getUseCompoundFile(). Why?
> And IndexWriter.addIndexes(IndexReader...) queries instanceof LMP, instead
> of calling mp.useCompoundFile()?

getUseCompoundFile - is a setting on LMP.

MP.useCompoundFile is an actual API callback that should decide
whether to use compounding or not on per-segment basis. For LMP is
just returns the value of getUseCompoundFile (that is, until Mike's
patch that switches off compounding for large segments).

IW.addIndexes should use mp.useCompoundFile, I totally agree.

> So perhaps we should:
>
> 1) Fix IW to not case to LMP just to ask if it should create compound files
> or not. And then we can perhaps remove getLogMergePolicy from IW on 3x, and
> also removing the source for confusion.

You can't remove it on 3x, it's used by a host of deprecated methods
that access LMP's settings through IW.

> 2) Look at LMP and decide if there are method we believe can be placed on a
> general MP, such as mergeFactor or maxMergeDocs. LogMP is special in how it
> picks segments for merge - that is, log-based (levels). But maxMergeDocs,
> maxMergeSize, mergeFactor, are unrelated to log/levels. This is the sort of
> functionality I'd expect to find on a general MP impl.

Truly "general" MP impl should have no settings. Only callbacks that
tell IW how to behave in different cases.

What the hell is mergeFactor?! Right now this LMP setting serves two
completely separate roles at once.
1. it governs a maximum number of segments on a "level";
2. it also limits a maximum number of segments that can participate in
a single merge (thus throttling IO randomness).

Now why do you want to push that wreck of a setting onto unsuspecting
emerging MergePolicies?
Same goes to maxMergeWhatever. Why do you want to limit user's options
by forcing him to support these settings?


-- 
Kirill Zakharenko/Кирилл Захаренко (earwin@gmail.com)
Phone: +7 (495) 683-567-4
ICQ: 104465785

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message