lucene-solr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Au <bill.w...@gmail.com>
Subject Re: [jira] Commented: (SOLR-1513) Use Google Collections in ConcurrentLRUCache
Date Tue, 20 Oct 2009 13:46:34 GMT
+1 for having soft reference an available option by configuration, keeping
the current behavior as default.

Bill

2009/10/20 Noble Paul നോബിള്‍ नोब्ळ् <noble.paul@corp.aol.com>

> On Tue, Oct 20, 2009 at 6:07 PM, Mark Miller <markrmiller@gmail.com>
> wrote:
>
> > I'm +1 obviously ;) No one is talking about making it the default. And I
> > think its well known that soft value caches can be a valid choice -
> > thats why google has one in their collections here ;) Its a nice way to
> > let your cache grow and shrink based on the available RAM. Its not
> > always the right choice, but sure is a nice option. And it doesn't have
> > much to do with Lucene's FieldCaches. The main reason for a soft value
> > cache is not to avoid OOM. Set your cache sizes correctly for that. And
> > even if it was to avoid OOM, who cares if something else causes more of
> > them? Thats like not fixing a bug in a piece of code because another
> > piece of code has more bugs. Anyway, their purpose is to allow the cache
> > to size depending on the available free RAM IMO.
> >
> +1
>
> >
> > Noble Paul നോബിള്‍ नोब्ळ् wrote:
> > > So , is everyone now in favor of this feature? Who has a -1 on this?
> and
> > > what is the concern?
> > >
> > > On Tue, Oct 20, 2009 at 3:56 PM, Mark Miller <markrmiller@gmail.com>
> > wrote:
> > >
> > >
> > >> On Oct 20, 2009, at 12:12 AM, Shalin Shekhar Mangar <
> > >> shalinmangar@gmail.com> wrote:
> > >>
> > >>  I don't think the debate is about weak reference vs. soft references.
> > >>
> > >> There appears to be confusion between the two here no matter what the
> > >> debate - soft references are for cachinh, weak references are not so
> > much.
> > >> Getting it right is important.
> > >>
> > >>  I
> > >>
> > >>> guess the point that Lance is making is that using such a technique
> > will
> > >>> make application performance less predictable. There's also a good
> > chance
> > >>> that a soft reference based cache will cause cache thrashing and will
> > hide
> > >>> OOMs caused by inadequate cache sizes. So basically we trade an OOM
> for
> > >>> more
> > >>> CPU usage (due to re-computation of results).
> > >>>
> > >>>
> > >> That's the whole point. Your not hiding anything. I don't follow you.
> > >>
> > >>
> > >>
> > >>
> > >>> Personally, I think giving an option is fine. What if the user does
> not
> > >>> have
> > >>> enough RAM and he is willing to pay the price? Right now, there is
no
> > way
> > >>> he
> > >>> can do that at all. However, the most frequent reason behind OOMs is
> > not
> > >>> having enough RAM to create the field caches and not Solr caches, so
> > I'm
> > >>> not
> > >>> sure how important this is.
> > >>>
> > >>>
> > >> How important is any feature? You don't have a use for it, so it's not
> > >> important to you - someone else does so it is important to them. Soft
> > value
> > >> caches can be useful.
> > >>
> > >>
> > >>
> > >>
> > >>> On Tue, Oct 20, 2009 at 8:41 AM, Mark Miller <markrmiller@gmail.com>
> > >>> wrote:
> > >>>
> > >>>  There is a difference - weak references are not for very good for
> > caches
> > >>>
> > >>>> -
> > >>>> soft references (soft values here) are good for caches in most
jvms.
> > They
> > >>>> can be very nice. Weak refs are eagerly reclaimed - it's suggested
> > that
> > >>>> impls should not eagerly reclaim soft refs.
> > >>>>
> > >>>> - Mark
> > >>>>
> > >>>> http://www.lucidimagination.com (mobile)
> > >>>>
> > >>>>
> > >>>> On Oct 19, 2009, at 8:22 PM, Lance Norskog <goksron@gmail.com>
> wrote:
> > >>>>
> > >>>> "Soft references" then. "Weak pointers" is an older term. (They're
> > >>>>
> > >>>>
> > >>>>> "weak" because some bully can steal their candy.)
> > >>>>>
> > >>>>> On Sun, Oct 18, 2009 at 8:37 PM, Jason Rutherglen
> > >>>>> <jason.rutherglen@gmail.com> wrote:
> > >>>>>
> > >>>>>  Lance,
> > >>>>>
> > >>>>>> Do you mean soft references?
> > >>>>>>
> > >>>>>> On Sun, Oct 18, 2009 at 3:59 PM, Lance Norskog <goksron@gmail.com
> >
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>  -1 for weak references in caching.
> > >>>>>>
> > >>>>>>> This makes memory management less deterministic (predictable)
and
> > at
> > >>>>>>> peak can cause cache-thrashing. In other words, the
worst case
> gets
> > >>>>>>> even more worse. When designing a system I want predictability
> and
> > I
> > >>>>>>> want to control the worst case, because system meltdowns
are
> caused
> > by
> > >>>>>>> the worst case. Having thousands of small weak references
does
> the
> > >>>>>>> opposite.
> > >>>>>>>
> > >>>>>>> On Sat, Oct 17, 2009 at 2:00 AM, Noble Paul (JIRA)
<
> > jira@apache.org>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>>  [
> > >>>>>>>>
> > >>>>>>>>
> >
> https://issues.apache.org/jira/browse/SOLR-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766864#action_12766864
> > >>>>>>>> ]
> > >>>>>>>>
> > >>>>>>>> Noble Paul commented on SOLR-1513:
> > >>>>>>>> ----------------------------------
> > >>>>>>>>
> > >>>>>>>> bq.Google Collections is already checked in as
a dependency of
> > Carrot
> > >>>>>>>> clustering.
> > >>>>>>>>
> > >>>>>>>> in that e need to move it to core.
> > >>>>>>>>
> > >>>>>>>> Jason . We do not need to remove the original option.
We can
> > probably
> > >>>>>>>> add an extra parameter say softRef="true" or something.
That way
> ,
> > we
> > >>>>>>>> are
> > >>>>>>>> not screwing up anything and perf benefits can
be studied
> > separately.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Use Google Collections in ConcurrentLRUCache
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> --------------------------------------------
> > >>>>>>>>>
> > >>>>>>>>>              Key: SOLR-1513
> > >>>>>>>>>              URL:
> > https://issues.apache.org/jira/browse/SOLR-1513
> > >>>>>>>>>          Project: Solr
> > >>>>>>>>>       Issue Type: Improvement
> > >>>>>>>>>       Components: search
> > >>>>>>>>>  Affects Versions: 1.4
> > >>>>>>>>>         Reporter: Jason Rutherglen
> > >>>>>>>>>         Priority: Minor
> > >>>>>>>>>          Fix For: 1.5
> > >>>>>>>>>
> > >>>>>>>>>      Attachments: google-collect-snapshot.jar,
SOLR-1513.patch
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> ConcurrentHashMap is used in ConcurrentLRUCache.
 The Google
> > >>>>>>>>> Colletions concurrent map implementation allows
for soft values
> > that
> > >>>>>>>>> are
> > >>>>>>>>> great for caches that potentially exceed the
allocated heap.
> >  Though
> > >>>>>>>>> I
> > >>>>>>>>> suppose Solr caches usually don't use too much
RAM?
> > >>>>>>>>> http://code.google.com/p/google-collections/
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>> --
> > >>>>>>>> This message is automatically generated by JIRA.
> > >>>>>>>> -
> > >>>>>>>> You can reply to this email to add a comment to
the issue
> online.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>> --
> > >>>>>>> Lance Norskog
> > >>>>>>> goksron@gmail.com
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>> --
> > >>>>> Lance Norskog
> > >>>>> goksron@gmail.com
> > >>>>>
> > >>>>>
> > >>>>>
> > >>> --
> > >>> Regards,
> > >>> Shalin Shekhar Mangar.
> > >>>
> > >>>
> > >
> > >
> > >
> >
> >
> > --
> > - Mark
> >
> > http://www.lucidimagination.com
> >
> >
> >
> >
>
>
> --
> -----------------------------------------------------
> Noble Paul | Principal Engineer| AOL | http://aol.com
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message