lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] Commented: (LUCENE-1052) Add an "termInfosIndexDivisor" to IndexReader
Date Tue, 20 Nov 2007 21:34:43 GMT


Michael McCandless commented on LUCENE-1052:

Maybe, instead, we should simply make it "easy" to subclass
TermInfosReader whenever a SegmentReader wants to instantiate it?

Ie, the formula is such an advanced use case that it seems appropriate
to subclass instead of trying to break it out into a special
interface/abstract class?

Of course, we need to know this class at SegmentReader construction
time, so I think to specify it we should in fact take Doug's suggested
approach using generic properties.

The challenge with Lucene (and Hadoop) is how can you reach deep down
into a complex static method call to change various
details of the embedded *Readers while they are being constructed,
and, after they are constructed... I agree it is messy now that we
must propogate the setTermInfosIndexInterval method up the *Reader
hierarchy when not all Readers would even use a TermInfosReader.

So ... maybe we 1) implement generic Lucene properties w/ static
classes/methods to set/get these properties, then 2) remove
set/getTermInfosIndexInterval from *Reader and make a generic property
for it instead, and 3) add another property that allows you to specify
the Class (or String name) of that is your TermInfosReader subclass
(and make it non-final)?

> Add an "termInfosIndexDivisor" to IndexReader
> ---------------------------------------------
>                 Key: LUCENE-1052
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.2
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 2.3
>         Attachments: LUCENE-1052.patch, termInfosConfigurer.patch
> The termIndexInterval, set during indexing time, let's you tradeoff
> how much RAM is used by a reader to load the indexed terms vs cost of
> seeking to the specific term you want to load.
> But the downside is you must set it at indexing time.
> This issue adds an indexDivisor to TermInfosReader so that on opening
> a reader you could further sub-sample the the termIndexInterval to use
> less RAM.  EG a setting of 2 means every 2 * termIndexInterval is
> loaded into RAM.
> This is particularly useful if your index has a great many terms (eg
> you accidentally indexed binary terms).
> Spinoff from this thread:

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message