lucene-solr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: [jira] Commented: (SOLR-1513) Use Google Collections in ConcurrentLRUCache
Date Tue, 20 Oct 2009 12:37:21 GMT
I'm +1 obviously ;) No one is talking about making it the default. And I
think its well known that soft value caches can be a valid choice -
thats why google has one in their collections here ;) Its a nice way to
let your cache grow and shrink based on the available RAM. Its not
always the right choice, but sure is a nice option. And it doesn't have
much to do with Lucene's FieldCaches. The main reason for a soft value
cache is not to avoid OOM. Set your cache sizes correctly for that. And
even if it was to avoid OOM, who cares if something else causes more of
them? Thats like not fixing a bug in a piece of code because another
piece of code has more bugs. Anyway, their purpose is to allow the cache
to size depending on the available free RAM IMO.

Noble Paul നോബിള്‍ नोब्ळ् wrote:
> So , is everyone now in favor of this feature? Who has a -1 on this? and
> what is the concern?
>
> On Tue, Oct 20, 2009 at 3:56 PM, Mark Miller <markrmiller@gmail.com> wrote:
>
>   
>> On Oct 20, 2009, at 12:12 AM, Shalin Shekhar Mangar <
>> shalinmangar@gmail.com> wrote:
>>
>>  I don't think the debate is about weak reference vs. soft references.
>>     
>> There appears to be confusion between the two here no matter what the
>> debate - soft references are for cachinh, weak references are not so much.
>> Getting it right is important.
>>
>>  I
>>     
>>> guess the point that Lance is making is that using such a technique will
>>> make application performance less predictable. There's also a good chance
>>> that a soft reference based cache will cause cache thrashing and will hide
>>> OOMs caused by inadequate cache sizes. So basically we trade an OOM for
>>> more
>>> CPU usage (due to re-computation of results).
>>>
>>>       
>> That's the whole point. Your not hiding anything. I don't follow you.
>>
>>
>>
>>     
>>> Personally, I think giving an option is fine. What if the user does not
>>> have
>>> enough RAM and he is willing to pay the price? Right now, there is no way
>>> he
>>> can do that at all. However, the most frequent reason behind OOMs is not
>>> having enough RAM to create the field caches and not Solr caches, so I'm
>>> not
>>> sure how important this is.
>>>
>>>       
>> How important is any feature? You don't have a use for it, so it's not
>> important to you - someone else does so it is important to them. Soft value
>> caches can be useful.
>>
>>
>>
>>     
>>> On Tue, Oct 20, 2009 at 8:41 AM, Mark Miller <markrmiller@gmail.com>
>>> wrote:
>>>
>>>  There is a difference - weak references are not for very good for caches
>>>       
>>>> -
>>>> soft references (soft values here) are good for caches in most jvms. They
>>>> can be very nice. Weak refs are eagerly reclaimed - it's suggested that
>>>> impls should not eagerly reclaim soft refs.
>>>>
>>>> - Mark
>>>>
>>>> http://www.lucidimagination.com (mobile)
>>>>
>>>>
>>>> On Oct 19, 2009, at 8:22 PM, Lance Norskog <goksron@gmail.com> wrote:
>>>>
>>>> "Soft references" then. "Weak pointers" is an older term. (They're
>>>>
>>>>         
>>>>> "weak" because some bully can steal their candy.)
>>>>>
>>>>> On Sun, Oct 18, 2009 at 8:37 PM, Jason Rutherglen
>>>>> <jason.rutherglen@gmail.com> wrote:
>>>>>
>>>>>  Lance,
>>>>>           
>>>>>> Do you mean soft references?
>>>>>>
>>>>>> On Sun, Oct 18, 2009 at 3:59 PM, Lance Norskog <goksron@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>  -1 for weak references in caching.
>>>>>>             
>>>>>>> This makes memory management less deterministic (predictable)
and at
>>>>>>> peak can cause cache-thrashing. In other words, the worst case
gets
>>>>>>> even more worse. When designing a system I want predictability
and I
>>>>>>> want to control the worst case, because system meltdowns are
caused by
>>>>>>> the worst case. Having thousands of small weak references does
the
>>>>>>> opposite.
>>>>>>>
>>>>>>> On Sat, Oct 17, 2009 at 2:00 AM, Noble Paul (JIRA) <jira@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>>>>  [
>>>>>>>>
>>>>>>>> https://issues.apache.org/jira/browse/SOLR-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766864#action_12766864
>>>>>>>> ]
>>>>>>>>
>>>>>>>> Noble Paul commented on SOLR-1513:
>>>>>>>> ----------------------------------
>>>>>>>>
>>>>>>>> bq.Google Collections is already checked in as a dependency
of Carrot
>>>>>>>> clustering.
>>>>>>>>
>>>>>>>> in that e need to move it to core.
>>>>>>>>
>>>>>>>> Jason . We do not need to remove the original option. We
can probably
>>>>>>>> add an extra parameter say softRef="true" or something. That
way , we
>>>>>>>> are
>>>>>>>> not screwing up anything and perf benefits can be studied
separately.
>>>>>>>>
>>>>>>>>
>>>>>>>> Use Google Collections in ConcurrentLRUCache
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> --------------------------------------------
>>>>>>>>>
>>>>>>>>>              Key: SOLR-1513
>>>>>>>>>              URL: https://issues.apache.org/jira/browse/SOLR-1513
>>>>>>>>>          Project: Solr
>>>>>>>>>       Issue Type: Improvement
>>>>>>>>>       Components: search
>>>>>>>>>  Affects Versions: 1.4
>>>>>>>>>         Reporter: Jason Rutherglen
>>>>>>>>>         Priority: Minor
>>>>>>>>>          Fix For: 1.5
>>>>>>>>>
>>>>>>>>>      Attachments: google-collect-snapshot.jar, SOLR-1513.patch
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ConcurrentHashMap is used in ConcurrentLRUCache.  The
Google
>>>>>>>>> Colletions concurrent map implementation allows for soft
values that
>>>>>>>>> are
>>>>>>>>> great for caches that potentially exceed the allocated
heap.  Though
>>>>>>>>> I
>>>>>>>>> suppose Solr caches usually don't use too much RAM?
>>>>>>>>> http://code.google.com/p/google-collections/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>> --
>>>>>>>> This message is automatically generated by JIRA.
>>>>>>>> -
>>>>>>>> You can reply to this email to add a comment to the issue
online.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>> --
>>>>>>> Lance Norskog
>>>>>>> goksron@gmail.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>> --
>>>>> Lance Norskog
>>>>> goksron@gmail.com
>>>>>
>>>>>
>>>>>           
>>> --
>>> Regards,
>>> Shalin Shekhar Mangar.
>>>
>>>       
>
>
>   


-- 
- Mark

http://www.lucidimagination.com




Mime
View raw message