lucene-solr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shalin Shekhar Mangar <shalinman...@gmail.com>
Subject Re: [jira] Commented: (SOLR-1513) Use Google Collections in ConcurrentLRUCache
Date Tue, 20 Oct 2009 04:12:22 GMT
I don't think the debate is about weak reference vs. soft references. I
guess the point that Lance is making is that using such a technique will
make application performance less predictable. There's also a good chance
that a soft reference based cache will cause cache thrashing and will hide
OOMs caused by inadequate cache sizes. So basically we trade an OOM for more
CPU usage (due to re-computation of results).

Personally, I think giving an option is fine. What if the user does not have
enough RAM and he is willing to pay the price? Right now, there is no way he
can do that at all. However, the most frequent reason behind OOMs is not
having enough RAM to create the field caches and not Solr caches, so I'm not
sure how important this is.

On Tue, Oct 20, 2009 at 8:41 AM, Mark Miller <markrmiller@gmail.com> wrote:

> There is a difference - weak references are not for very good for caches -
> soft references (soft values here) are good for caches in most jvms. They
> can be very nice. Weak refs are eagerly reclaimed - it's suggested that
> impls should not eagerly reclaim soft refs.
>
> - Mark
>
> http://www.lucidimagination.com (mobile)
>
>
> On Oct 19, 2009, at 8:22 PM, Lance Norskog <goksron@gmail.com> wrote:
>
>  "Soft references" then. "Weak pointers" is an older term. (They're
>> "weak" because some bully can steal their candy.)
>>
>> On Sun, Oct 18, 2009 at 8:37 PM, Jason Rutherglen
>> <jason.rutherglen@gmail.com> wrote:
>>
>>> Lance,
>>>
>>> Do you mean soft references?
>>>
>>> On Sun, Oct 18, 2009 at 3:59 PM, Lance Norskog <goksron@gmail.com>
>>> wrote:
>>>
>>>> -1 for weak references in caching.
>>>>
>>>> This makes memory management less deterministic (predictable) and at
>>>> peak can cause cache-thrashing. In other words, the worst case gets
>>>> even more worse. When designing a system I want predictability and I
>>>> want to control the worst case, because system meltdowns are caused by
>>>> the worst case. Having thousands of small weak references does the
>>>> opposite.
>>>>
>>>> On Sat, Oct 17, 2009 at 2:00 AM, Noble Paul (JIRA) <jira@apache.org>
>>>> wrote:
>>>>
>>>>>
>>>>>   [
>>>>> https://issues.apache.org/jira/browse/SOLR-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766864#action_12766864
>>>>>  ]
>>>>>
>>>>> Noble Paul commented on SOLR-1513:
>>>>> ----------------------------------
>>>>>
>>>>> bq.Google Collections is already checked in as a dependency of Carrot
>>>>> clustering.
>>>>>
>>>>> in that e need to move it to core.
>>>>>
>>>>> Jason . We do not need to remove the original option. We can probably
>>>>> add an extra parameter say softRef="true" or something. That way , we
are
>>>>> not screwing up anything and perf benefits can be studied separately.
>>>>>
>>>>>
>>>>>  Use Google Collections in ConcurrentLRUCache
>>>>>> --------------------------------------------
>>>>>>
>>>>>>                Key: SOLR-1513
>>>>>>                URL: https://issues.apache.org/jira/browse/SOLR-1513
>>>>>>            Project: Solr
>>>>>>         Issue Type: Improvement
>>>>>>         Components: search
>>>>>>   Affects Versions: 1.4
>>>>>>           Reporter: Jason Rutherglen
>>>>>>           Priority: Minor
>>>>>>            Fix For: 1.5
>>>>>>
>>>>>>        Attachments: google-collect-snapshot.jar, SOLR-1513.patch
>>>>>>
>>>>>>
>>>>>> ConcurrentHashMap is used in ConcurrentLRUCache.  The Google
>>>>>> Colletions concurrent map implementation allows for soft values that
are
>>>>>> great for caches that potentially exceed the allocated heap.  Though
I
>>>>>> suppose Solr caches usually don't use too much RAM?
>>>>>> http://code.google.com/p/google-collections/
>>>>>>
>>>>>
>>>>> --
>>>>> This message is automatically generated by JIRA.
>>>>> -
>>>>> You can reply to this email to add a comment to the issue online.
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Lance Norskog
>>>> goksron@gmail.com
>>>>
>>>>
>>>
>>
>>
>> --
>> Lance Norskog
>> goksron@gmail.com
>>
>


-- 
Regards,
Shalin Shekhar Mangar.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message