lucene-solr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: [jira] Commented: (SOLR-1513) Use Google Collections in ConcurrentLRUCache
Date Tue, 20 Oct 2009 10:26:19 GMT


On Oct 20, 2009, at 12:12 AM, Shalin Shekhar Mangar <shalinmangar@gmail.com 
 > wrote:

> I don't think the debate is about weak reference vs. soft references.

There appears to be confusion between the two here no matter what the  
debate - soft references are for cachinh, weak references are not so  
much. Getting it right is important.

> I
> guess the point that Lance is making is that using such a technique  
> will
> make application performance less predictable. There's also a good  
> chance
> that a soft reference based cache will cause cache thrashing and  
> will hide
> OOMs caused by inadequate cache sizes. So basically we trade an OOM  
> for more
> CPU usage (due to re-computation of results).

That's the whole point. Your not hiding anything. I don't follow you.


>
> Personally, I think giving an option is fine. What if the user does  
> not have
> enough RAM and he is willing to pay the price? Right now, there is  
> no way he
> can do that at all. However, the most frequent reason behind OOMs is  
> not
> having enough RAM to create the field caches and not Solr caches, so  
> I'm not
> sure how important this is.

How important is any feature? You don't have a use for it, so it's not  
important to you - someone else does so it is important to them. Soft  
value caches can be useful.

>
> On Tue, Oct 20, 2009 at 8:41 AM, Mark Miller <markrmiller@gmail.com>  
> wrote:
>
>> There is a difference - weak references are not for very good for  
>> caches -
>> soft references (soft values here) are good for caches in most  
>> jvms. They
>> can be very nice. Weak refs are eagerly reclaimed - it's suggested  
>> that
>> impls should not eagerly reclaim soft refs.
>>
>> - Mark
>>
>> http://www.lucidimagination.com (mobile)
>>
>>
>> On Oct 19, 2009, at 8:22 PM, Lance Norskog <goksron@gmail.com> wrote:
>>
>> "Soft references" then. "Weak pointers" is an older term. (They're
>>> "weak" because some bully can steal their candy.)
>>>
>>> On Sun, Oct 18, 2009 at 8:37 PM, Jason Rutherglen
>>> <jason.rutherglen@gmail.com> wrote:
>>>
>>>> Lance,
>>>>
>>>> Do you mean soft references?
>>>>
>>>> On Sun, Oct 18, 2009 at 3:59 PM, Lance Norskog <goksron@gmail.com>
>>>> wrote:
>>>>
>>>>> -1 for weak references in caching.
>>>>>
>>>>> This makes memory management less deterministic (predictable)  
>>>>> and at
>>>>> peak can cause cache-thrashing. In other words, the worst case  
>>>>> gets
>>>>> even more worse. When designing a system I want predictability  
>>>>> and I
>>>>> want to control the worst case, because system meltdowns are  
>>>>> caused by
>>>>> the worst case. Having thousands of small weak references does the
>>>>> opposite.
>>>>>
>>>>> On Sat, Oct 17, 2009 at 2:00 AM, Noble Paul (JIRA) <jira@apache.org

>>>>> >
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>  [
>>>>>> https://issues.apache.org/jira/browse/SOLR-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766864#action_12766864
>>>>>> ]
>>>>>>
>>>>>> Noble Paul commented on SOLR-1513:
>>>>>> ----------------------------------
>>>>>>
>>>>>> bq.Google Collections is already checked in as a dependency of  
>>>>>> Carrot
>>>>>> clustering.
>>>>>>
>>>>>> in that e need to move it to core.
>>>>>>
>>>>>> Jason . We do not need to remove the original option. We can  
>>>>>> probably
>>>>>> add an extra parameter say softRef="true" or something. That  
>>>>>> way , we are
>>>>>> not screwing up anything and perf benefits can be studied  
>>>>>> separately.
>>>>>>
>>>>>>
>>>>>> Use Google Collections in ConcurrentLRUCache
>>>>>>> --------------------------------------------
>>>>>>>
>>>>>>>               Key: SOLR-1513
>>>>>>>               URL: https://issues.apache.org/jira/browse/SOLR-1513
>>>>>>>           Project: Solr
>>>>>>>        Issue Type: Improvement
>>>>>>>        Components: search
>>>>>>>  Affects Versions: 1.4
>>>>>>>          Reporter: Jason Rutherglen
>>>>>>>          Priority: Minor
>>>>>>>           Fix For: 1.5
>>>>>>>
>>>>>>>       Attachments: google-collect-snapshot.jar, SOLR-1513.patch
>>>>>>>
>>>>>>>
>>>>>>> ConcurrentHashMap is used in ConcurrentLRUCache.  The Google
>>>>>>> Colletions concurrent map implementation allows for soft  
>>>>>>> values that are
>>>>>>> great for caches that potentially exceed the allocated heap.
  
>>>>>>> Though I
>>>>>>> suppose Solr caches usually don't use too much RAM?
>>>>>>> http://code.google.com/p/google-collections/
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> This message is automatically generated by JIRA.
>>>>>> -
>>>>>> You can reply to this email to add a comment to the issue online.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Lance Norskog
>>>>> goksron@gmail.com
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Lance Norskog
>>> goksron@gmail.com
>>>
>>
>
>
> -- 
> Regards,
> Shalin Shekhar Mangar.

Mime
View raw message