lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] Commented: (LUCENE-831) Complete overhaul of FieldCache API/Implementation
Date Fri, 27 Mar 2009 09:46:51 GMT


Michael McCandless commented on LUCENE-831:

I'd like to see the new FieldCache API de-emphasize "get me a single array holding all values
for all docs in the index" for a MultiReader. That invocation is exceptionally costly in the
context of reopened readers, and providing the illusion that one can simply get this array
is dangerous. It's a "leaky API", like how virtual memory API pretends you can use more memory
than is physically available. 

I think it's OK to return an array-of-arrays (ie, one contiguous array per underlying segment);
if the app really wants to make a massive array & concatenate it, they can do so outside
of the FieldCache API. 

Or an method you call to get a given doc's value (like DocValues in
Or an iterator API to step through all the values. 

We should also set this API up as much as possible for LUCENE-1231. Ie, the current "un-invert
the field" approach that FieldCache takes is merely one source of values per doc. Column stride
fields in the future will be a different (faster) source of values, that should be able to
"just plug in" under the hood somehow to this same exposure API. 

On Uwe's suggestion for some flexibility on how the un-inversion takes place, I think allowing
differing degrees of extension makes sense. EG we already allow you to provide a custom parser.
We need to allow control on whether a given value replaces the already-seen value (LUCENE-1372),
or whether to stop the looping early (Uwe's needs for improving Trie). We should also allow
outright entire custom class that creates the value array.

> Complete overhaul of FieldCache API/Implementation
> --------------------------------------------------
>                 Key: LUCENE-831
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Search
>            Reporter: Hoss Man
>             Fix For: 3.0
>         Attachments:, fieldcache-overhaul.032208.diff, fieldcache-overhaul.diff,
fieldcache-overhaul.diff, LUCENE-831.03.28.2008.diff, LUCENE-831.03.30.2008.diff, LUCENE-831.03.31.2008.diff,
LUCENE-831.patch, LUCENE-831.patch, LUCENE-831.patch, LUCENE-831.patch, LUCENE-831.patch,
> Motivation:
> 1) Complete overhaul the API/implementation of "FieldCache" type things...
>     a) eliminate global static map keyed on IndexReader (thus
>         eliminating synch block between completley independent IndexReaders)
>     b) allow more customization of cache management (ie: use 
>         expiration/replacement strategies, disk backed caches, etc)
>     c) allow people to define custom cache data logic (ie: custom
>         parsers, complex datatypes, etc... anything tied to a reader)
>     d) allow people to inspect what's in a cache (list of CacheKeys) for
>         an IndexReader so a new IndexReader can be likewise warmed. 
>     e) Lend support for smarter cache management if/when
>         IndexReader.reopen is added (merging of cached data from subReaders).
> 2) Provide backwards compatibility to support existing FieldCache API with
>     the new implementation, so there is no redundent caching as client code
>     migrades to new API.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message