lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Busch <>
Subject Re: Does LUCENE-831) "Complete overhaul of FieldCache API" provide fieldcache offloading to disk?
Date Fri, 18 Apr 2008 12:54:46 GMT
Michael McCandless wrote:
> OK so in this approach, a CSF is an "on disk" format, while the 
> FieldCache represents loading all (or maybe eventually subsets as 
> controlled by a cache policy) into a memory cache.  And since they both 
> implement FSV you can swap either in when you need it.

Yes, exactly.


> This sounds good!
> Mike
> Michael Busch wrote:
>> Chris Hostetter wrote:
>>> : But then the FieldCache is just starting to feel alot like 
>>> column-stride
>>> : fields
>>> : (LUCENE-1231).
>>> that's what i've been thinking ... my goal with LUCENE-831 was to 
>>> make it easier to manage FieldCache and hopefully the norms[] as well 
>>> particularly in the case of reopen ... but with column-stride fields 
>>> the need for both of those might go away completely)
>> (moved to java-dev, java-user cc'd)
>> My goal is it not to get rid of the FieldCache by adding column-stride 
>> fields (CSF), but instead to make them the default source for the 
>> FieldCache.
>> We should introduce an interface, named maybe FieldValueSource, that 
>> the new FieldCache implements, and also the CSF API. That has some 
>> advantages:
>> - Norms can be stored as CSF, and can be accessed using the 
>> FieldValueSource API. Then we can easily add an option to IndexReader 
>> whether to cache norms in memory (i. e. the new FieldCache) or not. 
>> When users have huge indexes on 32bit machines, where the norms would 
>> consume too much memory, they can disable caching them, of course 
>> search performance will suffer (but that's better than OutOfMemoryErrors)
>> - The function queries can use the FieldValueSource interface to 
>> retrieve the values (allowing us to get rid of function/ValueSource).
>> - Any consumer of the FieldValueSource does not have to care about 
>> whether or not values are cached and how. If performance is too slow 
>> and memory permits, caching can be enabled very easily.
>> - We will still support loading the fieldcache from the dictionary for 
>> backwards compatibility, but we should think about deprecating this 
>> and eventually get rid of it. We probably shouldn't add an 
>> implementation of FieldValueSource that reads from the dictionary, 
>> because performance would be terrible in the non-cached mode.
>> -Michael
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message