Mailing-List: contact solr-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-dev@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of markrmiller@gmail.com
 designates 209.85.221.189 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:from:to:in-reply-to:content-type
         :content-transfer-encoding:x-mailer:mime-version:subject:date
         :references;
        b=fA8AojjBv6h6FCyXQc9f4CZ65Q6SPlz3P7QWezbUsIwYSBffFbWyT6ReniNCenURIC
         CUaGimU+TdG1tbkwvc/7jzNZ6njXlTlV+CFaofNRhBW/fUl5T12zhYHT/v/gSqkcddeD
         zABJGyq/C9Rd6k3IPmj3KyeJGDDW72Y/djESo=
Message-Id: <CBFF7EB6-9563-494A-81D5-315E4DFC78CD@gmail.com>
From: Mark Miller <markrmiller@gmail.com>
To: "solr-dev@lucene.apache.org" <solr-dev@lucene.apache.org>
In-Reply-To: <5e76b0ad0910212236w10450342yde11c32980bc9e11@mail.gmail.com>
Content-Type: text/plain;
	charset=utf-8;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (iPhone Mail 7D11)
Subject: Re: [jira] Commented: (SOLR-1513) Use Google Collections in
 ConcurrentLRUCache
Date: Thu, 22 Oct 2009 07:35:52 -0400
References: <1638194484.1255646611371.JavaMail.jira@brutus>
 <87c998320910181559m7202d7a1ydf2f9bd56e721d4f@mail.gmail.com>
 <85d3c3b60910182037jae56123l91ced47f53655675@mail.gmail.com>
 <87c998320910191722s38070638ib6ad487458b7ee50@mail.gmail.com>
 <7B0CA961-999D-4ED1-847F-5CAD3F2D0F34@gmail.com>
 <69de18140910192112l5bb52070rc4066d464e6f4dec@mail.gmail.com>
 <34319782-E169-4887-B361-E989A9694648@gmail.com>
 <69de18140910201127y283296eaye16b8ff581beb843@mail.gmail.com>
 <87c998320910201704n1e5dba2awa9237e6eecf73f57@mail.gmail.com>
 <4ADF06F1.2080809@gmail.com>
 <5e76b0ad0910212236w10450342yde11c32980bc9e11@mail.gmail.com>

I'm not against that.

- Mark

http://www.lucidimagination.com (mobile)

On Oct 22, 2009, at 1:36 AM, Noble Paul =E0=B4=A8=E0=B5=8B=E0=B4=AC=E0=B4=BF=
=E0=B4=B3=E0=B5=8D=E2=80=8D  =E0=A4=A8=E0=A5=8B=20
=E0=A4=AC=E0=A5=8D=E0=A4=B3=E0=A5=8D <noble.paul@corp.aol.com> wrote:

> On Wed, Oct 21, 2009 at 6:34 PM, Mark Miller <markrmiller@gmail.com> =20=

> wrote:
>> bq.  and Mark is representing "just keep working, ok?".
>>
>> But I'm not :) Like I said, I don't view the purpose of a soft value
>> cache as avoiding OOM's. Size your caches correctly for that.
>>
>> For those that don't understand the how and why of soft value caches,
>> they probably should not choose to use it.
>
> Users may not have a clue on how much memory eventually the caches
> will take up. Now if the admin page can let them know cache trashing
> has happened , they can think of adding more RAM
>>
>> Lance Norskog wrote:
>>> On-topic: Will the Google implementations + soft references behave
>>> well with 8+ processors?
>>>
>>> Semi-on-topic: If you want to really know multiprocessor algorithms,
>>> this is the bible: "The Art Of Multiprocessor Programming". Hundreds
>>> of parallel algorithms for many different jobs, all coded in Java, =20=

>>> and
>>> cross-referenced with the java.util.concurrent package. Just =20
>>> amazing.
>>>
>>> =
http://www.elsevier.com/wps/find/bookdescription.cws_home/714091/descripti=
on#description
>>>
>>> Off-topic: I was representing a system troubleshooting philosophy:
>>> "Fail Early, Fail Loud". Meaning, if there is a problem like OOMs,
>>> tell me and I'll fix it permanently. But different situations call =20=

>>> for
>>> different answers, and Mark is representing "just keep working, =20
>>> ok?".
>>> Brittle v.s. Supple is one way to think of it.
>>>
>>> On Tue, Oct 20, 2009 at 11:27 AM, Shalin Shekhar Mangar
>>> <shalinmangar@gmail.com> wrote:
>>>
>>>> On Tue, Oct 20, 2009 at 3:56 PM, Mark Miller =20
>>>> <markrmiller@gmail.com> wrote:
>>>>
>>>>
>>>>> On Oct 20, 2009, at 12:12 AM, Shalin Shekhar Mangar <
>>>>> shalinmangar@gmail.com> wrote:
>>>>>
>>>>>  I don't think the debate is about weak reference vs. soft =20
>>>>> references.
>>>>>
>>>>> There appears to be confusion between the two here no matter =20
>>>>> what the
>>>>> debate - soft references are for cachinh, weak references are =20
>>>>> not so much.
>>>>> Getting it right is important.
>>>>>
>>>>>  I
>>>>>
>>>>>> guess the point that Lance is making is that using such a =20
>>>>>> technique will
>>>>>> make application performance less predictable. There's also a =20
>>>>>> good chance
>>>>>> that a soft reference based cache will cause cache thrashing =20
>>>>>> and will hide
>>>>>> OOMs caused by inadequate cache sizes. So basically we trade an =20=

>>>>>> OOM for
>>>>>> more
>>>>>> CPU usage (due to re-computation of results).
>>>>>>
>>>>>>
>>>>> That's the whole point. Your not hiding anything. I don't follow =20=

>>>>> you.
>>>>>
>>>>>
>>>> Using a soft reference based cache can hide the fact that one has =20=

>>>> inadequate
>>>> memory for the cache size one has configured. Don't get me wrong. =20=

>>>> I'm not
>>>> against the feature. I was merely trying to explain Lance's =20
>>>> concerns as I
>>>> understood them.
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>>> Personally, I think giving an option is fine. What if the user =20=

>>>>>> does not
>>>>>> have
>>>>>> enough RAM and he is willing to pay the price? Right now, there =20=

>>>>>> is no way
>>>>>> he
>>>>>> can do that at all. However, the most frequent reason behind =20
>>>>>> OOMs is not
>>>>>> having enough RAM to create the field caches and not Solr =20
>>>>>> caches, so I'm
>>>>>> not
>>>>>> sure how important this is.
>>>>>>
>>>>>>
>>>>> How important is any feature? You don't have a use for it, so =20
>>>>> it's not
>>>>> important to you - someone else does so it is important to them. =20=

>>>>> Soft value
>>>>> caches can be useful.
>>>>>
>>>> Don't jump to conclusions :)
>>>>
>>>> The reason behind this feature request is to have Solr caches =20
>>>> which resize
>>>> themselves when enough memory is not available. I agree that soft =20=

>>>> value
>>>> caches are useful for this. All I'm saying is that most OOMs that =20=

>>>> get
>>>> reported on the list are due to inadequate free memory for =20
>>>> allocating field
>>>> caches. Finding a way around that will be the key to make a =20
>>>> Lucene/Solr
>>>> application practical in a limited memory environment.
>>>>
>>>> Just for the record, I'm +1 for adding this feature but keeping =20
>>>> the current
>>>> behavior as the default.
>>>>
>>>> --
>>>> Regards,
>>>> Shalin Shekhar Mangar.
>>>>
>>>>
>>>
>>>
>>>
>>>
>>
>>
>> --
>> - Mark
>>
>> http://www.lucidimagination.com
>>
>>
>>
>>
>
>
>
> --=20
> -----------------------------------------------------
> Noble Paul | Principal Engineer| AOL | http://aol.com