lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Smith <>
Subject Re: IndexReader Cache - a different angle
Date Mon, 13 Sep 2010 13:05:19 GMT
  i created some time 
ago proposing pretty much what seems to be discussed here

  -- Tim

On 09/12/10 10:18, Simon Willnauer wrote:
> On Sun, Sep 12, 2010 at 11:46 AM, Michael McCandless
> <>  wrote:
>> Having hooks to enable an app to manage its own "external, private
>> stuff associated w/ each segment reader" would be useful and it's been
>> asked for in the past.  However, since we've now opened up
>> SegmentReader, SegmentInfo/s, etc., in recent releases, can't an app
>> already do this w/o core API changes?
> The visitor approach would simply be a little more than syntactic
> sugar where only new SubReader instances are passed to the callback.
> You can do the same with the already existing API like
> gatherSubReaders or getSequentialSubReaders. Every API I was talking
> about would just be simplification anyway and would be possible to
> build without changing the core.
>> I know Earwin has built a whole system like this on top of Lucene --
>> Earwin how did you do that...?  Did you make core changes to
>> Lucene...?
>> A custom Codec should be an excellent way to handle the specific use
>> cache (caching certain postings) -- by doing it as a Codec, any time
>> anything in Lucene needs to tap into that posting (query scorers,
>> filters, merging, applying deletes, etc), it hits this cache.  You
>> could model it like PulsingCodec, which wraps any other Codec but
>> handles the low-freq ones itself.  If you do it externally how would
>> core use of postings hit it?  (Or was that not the intention?)
>> I don't understand the filter use-case... the CachingWrapperFilter
>> already caches per-segment, so that reopen is efficient?  How would an
>> external cache (built on these hooks) be different?
> Man you are right - never mind :)
> simon
>> For faster filters we have to apply them like we do deleted docs if
>> the filter is "random access" (such as being cached), LUCENE-1536 --
>> flex actually makes this relatively easy now, since the postings API
>> no longer implicitly filters deleted docs (ie you provide your own
>> skipDocs) -- but these hooks won't fix that right?
>> Mike
>> On Sun, Sep 12, 2010 at 3:43 AM, Simon Willnauer
>> <>  wrote:
>>> Hey Shai,
>>> On Sun, Sep 12, 2010 at 6:51 AM, Shai Erera<>  wrote:
>>>> Hey Simon,
>>>> You're right that the application can develop a Caching mechanism outside
>>>> Lucene, and when reopen() is called, if it changed, iterate on the
>>>> sub-readers and init the Cache w/ the new ones.
>>> Alright, then we are on the same track I guess!
>>>> However, by building something like that inside Lucene, the application will
>>>> get more native support, and thus better performance, in some cases. For
>>>> example, consider a field fileType with 10 possible values, and for the sake
>>>> of simplicity, let's say that the index is divided evenly across them. Your
>>>> users always add such a term constraint to the query (e.g. they want to get
>>>> results of fileType:pdf or fileType:odt, and perhaps sometimes both, but
>>>> others). You have basically two ways of supporting this:
>>>> (1) Add such a term to the query / clause to a BooleanQuery w/ an AND
>>>> relation -- cons is that this term / posting is read for every query.
>>> Oh I wasn't saying that a cache framework would be obsolet and
>>> shouldn't be part of lucene. My intention was rather to generalize
>>> this functionality so that we can make the API change more easily and
>>> at the same time brining the infrastructure you are proposing in
>>> place.
>>> Regarding you example above, filters are a very good example where
>>> something like that could help to improve performance and we should
>>> provide it with lucene core but I would again prefer the least
>>> intrusive way to do so. If we can make that happen without adding any
>>> cache agnostic API we should do it. We really should try to sketch out
>>> a simple API with gives us access to the opened segReaders and see if
>>> that would be sufficient for our usecases. Specialization will always
>>> be possible but I doubt that it is needed.
>>>> (2) Write a Filter which works at the top IR level, that is refreshed
>>>> whenever the index is refreshed. This is better than (1), however has some
>>>> disadvantages:
>>>> (2.1) As Mike already proved (on some issue which I don't remember its
>>>> subject/number at the moment), if we could get Filter down to the lower
>>>> level components of Lucene's search, so e.g. it is used as the deleted docs
>>>> OBS, we can get better performance w/ Filters.
>>>> (2.2) The Filter is refreshed for the entire IR, and not just the changed
>>>> segments. Reason is, outside Collector, you have no way of telling
>>>> IndexSearcher "use Filter F1 for segment S1 and F2 for segment F2".
>>>> Loading/refreshing the Filter may be expensive, and definitely won't perform
>>>> well w/ NRT, where by definition you'd like to get small changes searchable
>>>> very fast.
>>> No doubt you are right about the above. A
>>> PerSegmentCachingFilterWrapper would be something we can easily do on
>>> an application level basis with the infrastructure we are talking
>>> about in place. While I don't exactly know how I feel that this
>>> particular problem should rather be addressed internally and I'm not
>>> sure if the high level Cache mechanism is the right way to do it but
>>> this is just a gut feeling. But when I think about it twice it might
>>> be way sufficient enough to do it....
>>>> Therefore I think that if we could provide the necessary hooks in Lucene,
>>>> let's call it a Cache plug-in for now, we can incrementally improve the
>>>> search process. I don't want to go too far into the design of a generic
>>>> plug-ins mechanism, but you're right (again :)) -- we could offer a
>>>> reopen(PluginProvider) which is entirely not about Cache, and Cache would
>>>> become one of the Plugins the PluginProvider provides. I just try to learn
>>>> from past experience -- when the discussion is focused, there's a better
>>>> chance of getting to a resolution. However if you think that in this case,
>>>> more generic API, as PluginProvider, would get us to a resolution faster,
>>>> don't mind spend some time to think about it. But for all practical
>>>> purposes, we should IMO start w/ a Cache plug-in, that is called like that,
>>>> and if it catches, generify it ...
>>> I absolutely agree the API might be more generic but our current
>>> use-case / PoC should be a caching. I don't like the name Plugin but
>>> thats a personal thing since you are not pluggin anything is.
>>> Something like SubreaderCallback or ReaderVisitor might be more
>>> accurate but lets argue about the details later. Why not sketching
>>> something out for the filter problem and follow on from there? The
>>> more iteration the better and back to your question if that would be
>>> something which could make it to be committable I would say if it
>>> works stand alone / not to tightly coupled I would absolutely say yes.
>>>> Unfortunately, I haven't had enough experience w/ Codecs yet (still on 3x)
>>>> so I can't comment on how feasible that solution is. I'll take your word
>>>> it that it's doable :). But this doesn't give us a 3x solution ... the
>>>> Caching framework on trunk can be developed w/ Codecs.
>>> I guess nobody really has except of mike and maybe one or two others
>>> but what I have done so far regarding codecs I would say that is the
>>> place to solve this particular problem. Maybe even lower than that on
>>> a Directory level. Anyhow, lets focus on application level caches for
>>> now. We are not aiming to provide a whole full fledged Cache API but
>>> the infrastructure to make it easier to build those on a app basis
>>> which would be a valuable improvement. We should also look at Solr's
>>> cache implementations and how they could benefit from this efforts
>>> since Solr uses app-level caching we can learn from API design wise.
>>> simon
>>>> Shai
>>>> On Sat, Sep 11, 2010 at 10:41 PM, Simon Willnauer
>>>> <>  wrote:
>>>>> Hi Shai,
>>>>> On Sat, Sep 11, 2010 at 8:08 PM, Shai Erera<> 
>>>>>> Hi
>>>>>> Lucene's Caches have been heavilydiscussed before (e.g., LUCENE-831,
>>>>>> LUCENE-2133 and LUCENE-2394) and from what I can tell, there have
>>>>>> many proposals to attack this problem, w/ no developed solution.
>>>>> I didn't go through those issues so forgive me if something I bring up
>>>>> has already been discussed.
>>>>> I have a couple of question about your proposal - please find them
>>>>> inline...
>>>>>> I'd like to explore a different, IMO much simpler, angle to attach
>>>>>> problem. Instead of having Lucene manage the Cache itself, we let
>>>>>> application manage it, however Lucene will provide the necessary
>>>>>> in IndexReader to allow it. The hooks I have in mind are:
>>>>>> (1) IndexReader current API for TermDocs, TermEnum, TermPositions
>>>>>> --
>>>>>> already exists.
>>>>>> (2) When reopen() is called, Lucene will take care to call a
>>>>>> Cache.load(IndexReader), so that the application can pull whatever
>>>>>> information
>>>>>> it needs from the passed-in IndexReader.
>>>>> Would that do anything else than passing the new reader (if reopened)
>>>>> to the caches load method? I wonder if this is more than
>>>>> If(newReader != oldReader)
>>>>>   Cache.load(newReader)
>>>>> If so something like that should be done on a segment reader anyway,
>>>>> right? From my perspective this isn't more than a callback or visitor
>>>>> that should walk though the subreaders and called for each reopened
>>>>> sub-reader. A cache-warming visitor / callback would then be trivial
>>>>> and the API would be more general.
>>>>>> So to be more concrete on my proposal, I'd like to support caching
>>>>>> the following way (and while I've spent some time thinking about
it, I'm
>>>>>> sure there are great suggestions to improve it):
>>>>>> * Application provides a CacheFactory to,
>>>>>> exposes some very simple API, such as createCache, or
>>>>>> initCache(IndexReader) etc. Something which returns a Cache object,
>>>>>> which does not have very strict/concrete API.
>>>>> My first question would be why the reader should know about Cache if
>>>>> there is no strict / concrete API?
>>>>> I can follow you with the CacheFactory to create cache objects but why
>>>>> would the reader have to know / "receive" this object? Maybe this is
>>>>> answered further down the path but I don't see the reason why the
>>>>> notion of a "cache" must exist within open/reopen or if that could be
>>>>> implemented in a more general "cache" - agnostic way.
>>>>>> * IndexReader, most probably at the SegmentReader level uses
>>>>>> CacheFactory to create a new Cache instance and calls its
>>>>>> load(IndexReader) method, so that the Cache would initialize itself.
>>>>> That is what I was thinking above - yet is that more than a callback
>>>>> for each reopened or opened segment reader?
>>>>>> * The application can use CacheFactory to obtain the Cache object
>>>>>> IndexReader (for example, during Collector.setNextReader), or we
>>>>>> have IndexReader offer a getCache() method.
>>>>> :)  until here the cache is only used by the application itself not by
>>>>> any Lucene API, right? I can think of many application specific data
>>>>> that could be useful to be associated with an IR beyond the cacheing
>>>>> use case - again this could be a more general API solving that
>>>>> problem.
>>>>>> * One of Cache API would be getCache(TYPE), where TYPE is a String
>>>>>> Object, or an interface CacheType w/ no methods, just to be a marker
>>>>>> one, and the application is free to impl it however it wants. That's
>>>>>> loose API, I know, but completely at the application hands, which
>>>>>> Lucene code simpler.
>>>>> I like the idea together with the metadata associating functionality
>>>>> from above something like public T IndexReader#get(Type<T>  type).
>>>>> Hmm that looks quiet similar to Attributes, does it?! :) However this
>>>>> could be done in many ways but again "cache" - agnositc
>>>>>> * We can introduce a TermsCache, TermEnumCache and TermVectorCache
>>>>>> provide the user w/ IndexReader-similar API, only more efficient
>>>>>> say TermDocs -- something w/ random access to the docs inside, perhaps
>>>>>> even an OpenBitSet. Lucene can take advantage of it if, say, we create
>>>>>> CachingSegmentReader which makes use of the cache, and checks every
>>>>>> termDocs() is called if the required Term is cached or not etc. I
>>>>>> I may be thinking too much ahead.
>>>>> I see what you are trying to do here. I also see how this could be
>>>>> useful but I guess coming up with a stable APi which serves lots of
>>>>> applications would be quiet hard. A CachingSegmentReader could be a
>>>>> very simple decorator which would not require to touch the IR
>>>>> interface. Something like that could be part of lucene but I'm not
>>>>> sure if necessarily part of lucene core.
>>>>>> That's more or less what I've been thinking. I'm sure there are many
>>>>>> details to iron out, but I hope I've managed to pass the general
>>>>>> proposal through to you.
>>>>> Absolutely, this is how it works isn't it!
>>>>>> What I'm after first, is to allow applications deal w/ postings caching
>>>>>> more
>>>>>> natively. For example, if you have a posting w/ payloads you'd like
>>>>>> read into memory, or if you would like a term's TermDocs to be cached
>>>>>> (to be used as a Filter) etc. -- instead of writing something that
>>>>>> work at the top IndexReader level, you'd be able to take advantage
>>>>>> Lucene internals, i.e. refresh the Cache only for the new segments
>>>>> I wonder if a custom codec would be the right place to implement
>>>>> caching / mem resident structures for Postings with payloads etc. You
>>>>> could do that on a higher level too but codec seems to be the way to
>>>>> go here, right?
>>>>> To utilize per segment capabilities a callback for (re)opened segment
>>>>> readers would be sufficient or do I miss something?
>>>>> simon
>>>>>> I'm sure that after this will be in place, we can refactor FieldCache
>>>>>> work w/ that API, perhaps as a Cache specific implementation. But
>>>>>> leave that for later.
>>>>>> I'd appreciate your comments. Before I set to implement it, I'd like
>>>>>> know if the idea has any chances of making it to a commit :).
>>>>>> Shai
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail:
>>>>> For additional commands, e-mail:
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail:
>>> For additional commands, e-mail:
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message