lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <>
Subject Re: Does LUCENE-831) "Complete overhaul of FieldCache API" provide fieldcache offloading to disk?
Date Fri, 18 Apr 2008 12:49:23 GMT

OK so in this approach, a CSF is an "on disk" format, while the  
FieldCache represents loading all (or maybe eventually subsets as  
controlled by a cache policy) into a memory cache.  And since they  
both implement FSV you can swap either in when you need it.

This sounds good!


Michael Busch wrote:
> Chris Hostetter wrote:
>> : But then the FieldCache is just starting to feel alot like  
>> column-stride
>> : fields
>> : (LUCENE-1231).
>> that's what i've been thinking ... my goal with LUCENE-831 was to  
>> make it easier to manage FieldCache and hopefully the norms[] as  
>> well particularly in the case of reopen ... but with column-stride  
>> fields the need for both of those might go away completely)
> (moved to java-dev, java-user cc'd)
> My goal is it not to get rid of the FieldCache by adding column- 
> stride fields (CSF), but instead to make them the default source  
> for the FieldCache.
> We should introduce an interface, named maybe FieldValueSource,  
> that the new FieldCache implements, and also the CSF API. That has  
> some advantages:
> - Norms can be stored as CSF, and can be accessed using the  
> FieldValueSource API. Then we can easily add an option to  
> IndexReader whether to cache norms in memory (i. e. the new  
> FieldCache) or not. When users have huge indexes on 32bit machines,  
> where the norms would consume too much memory, they can disable  
> caching them, of course search performance will suffer (but that's  
> better than OutOfMemoryErrors)
> - The function queries can use the FieldValueSource interface to  
> retrieve the values (allowing us to get rid of function/ValueSource).
> - Any consumer of the FieldValueSource does not have to care about  
> whether or not values are cached and how. If performance is too  
> slow and memory permits, caching can be enabled very easily.
> - We will still support loading the fieldcache from the dictionary  
> for backwards compatibility, but we should think about deprecating  
> this and eventually get rid of it. We probably shouldn't add an  
> implementation of FieldValueSource that reads from the dictionary,  
> because performance would be terrible in the non-cached mode.
> -Michael
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message