Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of jason.rutherglen@gmail.com
 designates 74.125.92.25 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=tkUOWdo+H9bE6hjEkKgENECQ/yQYS+TWmAJD0tzAXZAt/7uwp/guzgtPvxbW62kWbD
         tM6JevFsvXCO5azwAzXORirDUd5Ml/OorFpw8Ue+T3fakP4PbFib4wvA+X8bpDToE/GU
         XDumUcf87H/Op9Pcs+UJGB7CKqfZf43Ajb7vE=
MIME-Version: 1.0
In-Reply-To: <4AB91407.90705@lucidimagination.com>
References: <62AB8D44-ABF1-415C-B70E-8CE97B967EE2@mac.com>
	 <9ac0c6aa0909220748s5bf3e300m6e17f10c7e02bf94@mail.gmail.com>
	 <c68e39170909220808o727a8768i37f78cba59a6f1f5@mail.gmail.com>
	 <9ac0c6aa0909220837k7ec7ffafw903ac551dad13c5b@mail.gmail.com>
	 <85d3c3b60909220844h62978edfrb8f088bf7d7045a2@mail.gmail.com>
	 <9ac0c6aa0909220853h4edbcbf2t2d00cffbfdb7a2b3@mail.gmail.com>
	 <c68e39170909221038j1be9969dh28fdb5749f9ec0fa@mail.gmail.com>
	 <9ac0c6aa0909221042w7ff079e5o12c624612e5ac1c6@mail.gmail.com>
	 <4AB91102.1000001@attivio.com> <4AB91407.90705@lucidimagination.com>
Date: Tue, 22 Sep 2009 11:36:40 -0700
Message-ID: <85d3c3b60909221136s435c8802p983688b340bf5569@mail.gmail.com>
Subject: Re: How to leverage the LogMergePolicy "calibrateSizeByDeletes" patch
	in Solr ?
From: Jason Rutherglen <jason.rutherglen@gmail.com>
To: java-dev@lucene.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I have a working version of Simple FieldCache Merging LUCENE-1785 that
should go in real soon.

On Tue, Sep 22, 2009 at 11:14 AM, Mark Miller <markrmiller@gmail.com> wrote=
:
> 1. see IndexWriter and the method/class that Mike pointed out earlier
> for the warming.
>
> 2. See Lucene-831 - I think we will get some form of that in someday.
>
> Tim Smith wrote:
>> This sounds pretty interesting
>>
>> is there a proposed API for doing this warming yet?
>> Is there a ticket tracking this?
>>
>> for my use cases, it would be really nice for applications to be able
>> to associate a custom "IndexCache" object with an index reader, then
>> this pluggable "AutoWarmer" would be in charge of initializing this
>> cache for a segment reader. I have a number of caches outside the
>> realm of regular field caches that i associate with a segment,
>> currently doing this after getting the IndexReader by iterating over
>> its segments, and getting a cache object shared across all instances
>> of the same logical segment. it would be nice if i could just have my
>> "cache" object subclass a lucene IndexCache class and drop it right
>> into this auto warming infrastructure (would greatly simplify things).
>>
>> then, once the index reader has been closed, it would call close on
>> any attached IndexCache objects in order to free up memory/objects.
>> (so i don't have to maintain reference counts anymore)
>>
>> Seems this could also greatly simplify the current field caching
>> mechanisms, as the field caches could be associated with an
>> IndexReader directly using the attached "IndexCache" object, instead
>> of using static weak reference hash maps. (could then add methods like
>> getFieldCache() to the IndexReader)
>>
>> =A0-- Tim Smith
>>
>> Michael McCandless wrote:
>>> Well described, that's exactly it! =A0I like the concrete example :)
>>>
>>> Thanks Yonik.
>>>
>>> Mike
>>>
>>> On Tue, Sep 22, 2009 at 1:38 PM, Yonik Seeley
>>> <yonik@lucidimagination.com> wrote:
>>>
>>>> OK Mike, thanks for your patience - I understand now :-)
>>>>
>>>> Here's an example that helped me understand - hopefully it will add to
>>>> others understanding more than it confuses ;-)
>>>>
>>>> IW.getReader() =3D> segments=3D{A, B}
>>>> =A0// something causes a merge of A,B into AB to start
>>>> addDoc(doc1)
>>>> =A0// doc1 goes into segment C
>>>> IW.getReader() =3D> segments=3D{A, B, C}
>>>> =A0// merge isn't done yet, so getReader() still returns A,B instead o=
f
>>>> AB, but doc1 is still searchable!
>>>>
>>>> OK, in this scenario, there's no advantage to warming in the IW vs the=
 app.
>>>> Let's start over with a little different timing:
>>>>
>>>> segments=3D{A,B}
>>>> =A0// something causes a merge of A,B into AB to start
>>>> addDoc(doc1)
>>>> =A0// doc1 goes into segment C
>>>> =A0// merging of A,B into AB finishes
>>>> IW.getReader() =3D> segments=3D{AB, C}
>>>>
>>>> Oh, no... with warming at the app level, we need to warm the huge AB
>>>> segment before doc1 is visible. =A0We could continue using the old
>>>> reader while the warming is ongoing, so no user requests will
>>>> experience long queries, but doc1 isn't in the old segment.
>>>>
>>>> With warming in the IW (basically warming becomes part of the same
>>>> operation as merging), then getReader() would return segments=3D{A,B,C=
}
>>>> and doc1 would still be instantly searchable.
>>>>
>>>> The only way to duplicate this functionality at the app layer would be
>>>> to recognize that there is a new segment, try and figure out what old
>>>> segments were merged to create this new segment, and create a reader
>>>> that's a mix of old and new to avoid unwarmed segments - not nice.
>>>>
>>>> -Yonik
>>>> http://www.lucidimagination.com
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>
>>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org