lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Benge <todd.be...@gmail.com>
Subject Re: FieldCache Question
Date Wed, 04 Feb 2009 17:30:47 GMT
On Wed, Feb 4, 2009 at 10:01 AM, Mark Miller <markrmiller@gmail.com> wrote:
> Todd Benge wrote:
>>
>> Hi,
>>
>> I've been looking into the FieldCache API because of memory problems
>> we've been seeing in our production environment.
>>
>> We use various different sorts so over time the cache builds up and
>> servers stop responding.
>>
>> I decided to apply the patch for JIRA 831:
>>
>> https://issues.apache.org/jira/browse/LUCENE-831
>>
>
> This patch is likely to change quite a bit soon. A lot of what it was
doing
> has been taken care of by a separate issue. Whats likely to remain is just
> the API change, but even that is likely to change (an iterator api was
> discussed). So thats a volatile patch at the moment.
>>
>> so that I can switch out the caching to an LRU based cache.
>>
>> One thing I noticed in the code is that there is an array cache built
>> per IndexReader.  I'm wondering what the consequences would be if the
>> arrays were merged for all IndexReaders.  That would result in single
>> cached array for Strings, Floats, Ints, etc.
>>
>
> The original intent of this patch was almost to do the opposite if I am
> understanding you right. In any case, I am not sure why you think this
will
> help?

The intent is to reduce the amount of memory that is held in cache.  As it
is now, it looks like there is an array of comparators for each index
reader.  Most of the data in the array appears to be the same for each cache
so there is duplication for each type ( string, float).


>>
>> It looks like the arrays are used by the various comparators during
>> the sort but wasn't exactly sure so thought I'd ask what would happen
>> if I combined the arrays.
>>
>
> I wouldn't combine them. Wouldn't this give less granularity to your LRU
> cache anyway? That code is likely to change drastically, and I'm not sure
> what advantage you would see. Do you have an index so large that you can't
> supply the necessary RAM for the fieldcaches?
>
Yes - we're runnning about 80G in the indices so there's not enough RAM for
all the data in the fieldcache.

> How is your custom cache implementation working? Since you are trying to
> tweak things, it sounds like you are still having issues? I am not sure
how
> combining arrays would help you, but I don't understand what you are
trying
> to solve yet.
>
I actually have it working now by supplying a LRU cache map for each index
reader.  The only real difference between my implementation and lucene out
of box is the map storing the data is a LRU so it will force data out of the
cache.  The default implementation uses a HashMap that is tied to the
IndexReader so it never gets garbage collected and it's size will continue
to grow.

The problem I'm really trying to solve is stability in the servers.  I
realize that a LRU implementation will probably have some performance
degredation but wanted to try it out and compare.

> - Mark
>
>> Thanks,
>>
>> Todd
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message