Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of todd.benge@gmail.com
 designates 209.85.221.20 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=YpoZwN3kW6zHxiWcgdjdkglBUgqAi5b/r1q97wQFccZuF7CtfClnGala5ZrA7CM8q1
         iUoS8sNVryX46z+sz5oDV+zRWEKVOgWMXekPzrWutZKp39M+TQKiGKF6LwlpHXURX5IE
         +O2XO0KXwP3pUhiXKD1Sy56SMe9PsgGanZnAU=
MIME-Version: 1.0
In-Reply-To: <4989D35D.6040009@gmail.com>
References: <847931a70902040846r505638a9ub462d7db9b5f3125@mail.gmail.com>
	 <4989C9DD.80405@gmail.com>
	 <847931a70902040930y39d76478tf822ad24b2151926@mail.gmail.com>
	 <4989D35D.6040009@gmail.com>
Date: Wed, 4 Feb 2009 11:01:17 -0700
Message-ID: <847931a70902041001r4945bacele00c24962c070066@mail.gmail.com>
Subject: Re: FieldCache Question
From: Todd Benge <todd.benge@gmail.com>
To: java-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=0015175cb80608dcc904621b9322

--0015175cb80608dcc904621b9322
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On Wed, Feb 4, 2009 at 10:41 AM, Mark Miller <markrmiller@gmail.com> wrote:

> Todd Benge wrote:
>
>>
>> The intent is to reduce the amount of memory that is held in cache.  As it
>> is now, it looks like there is an array of comparators for each index
>> reader.  Most of the data in the array appears to be the same for each
>> cache
>> so there is duplication for each type ( string, float).
>>
>>
> Use an array cachekey and override it as not mergeable.
>
> I suppose in terms of the unuiqes terms array, you could see some
> duplication.
>
> I don't think there should be much duplication though - in the non String
> cases, each SegmentIndexReader will only hold the values for itself. The
> size of the sub arrays would be the same as the full array.
> In the String case, you will have duplicates for the unique terms array, so
> if you have a lot, that may cause issues, but the ordinal array will not be
> any larger. And the unuiqe terms array shouldnt be terrible - the number of
> terms per segment should drop logarithmically. I'm not sure you'll see much
> of a difference, and it would only be with String sorts.
>
> That is, unless you are creating your own separate FieldCaches on
> multisegmentreaders - then you would double everything.
>
>
>>  Yes - we're runnning about 80G in the indices so there's not enough RAM
>> for
>> all the data in the fieldcache.
>>
>>
> That is a large index. Can you share how many documents?


I don't have the exact number but I think it's 200 - 250 million documents.
I'll see if I can get some more realistic numbers and re-post.

Thanks for the help.

Todd


>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

--0015175cb80608dcc904621b9322--