lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: How to leverage the LogMergePolicy "calibrateSizeByDeletes" patch in Solr ?
Date Tue, 22 Sep 2009 18:11:55 GMT
On Tue, Sep 22, 2009 at 2:01 PM, Tim Smith <tsmith@attivio.com> wrote:

> is there a proposed API for doing this warming yet?

It's already committed and available in 2.9 (see
IndexWriter.setMergedSegmentWarmer).

> for my use cases, it would be really nice for applications to be able to
> associate a custom "IndexCache" object with an index reader, then this
> pluggable "AutoWarmer" would be in charge of initializing this cache for a
> segment reader. I have a number of caches outside the realm of regular field
> caches that i associate with a segment, currently doing this after getting
> the IndexReader by iterating over its segments, and getting a cache object
> shared across all instances of the same logical segment. it would be nice if
> i could just have my "cache" object subclass a lucene IndexCache class and
> drop it right into this auto warming infrastructure (would greatly simplify
> things).
>
> then, once the index reader has been closed, it would call close on any
> attached IndexCache objects in order to free up memory/objects. (so i don't
> have to maintain reference counts anymore)

Lucene doesn't expose this today; I think you have to track the
association externally.  But we could consider adding something like
this...

> Seems this could also greatly simplify the current field caching mechanisms,
> as the field caches could be associated with an IndexReader directly using
> the attached "IndexCache" object, instead of using static weak reference
> hash maps. (could then add methods like getFieldCache() to the IndexReader)

One challenge here is that there can easily be multiple SegmentReaders
"out there", for a single index segment.  EG if you reopen a reader
after new deletes are flushed to a previous segment.  In this case,
you'll have different SegmentReaders, but they intentionally share the
same entry in FieldCache since their "core" is shared.

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message