lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yonik Seeley <yo...@lucidimagination.com>
Subject Re: How to leverage the LogMergePolicy "calibrateSizeByDeletes" patch in Solr ?
Date Wed, 23 Sep 2009 05:15:32 GMT
Ya know, It turned out to be embarrassingly simple - I think I just
had a mental block from thinking about how Solr's warming worked for
so long.

Actually, it was so simple, yet I still got in wrong on the first
glance, that it reminded me of this:
http://www.marilynvossavant.com/forum/viewtopic.php?t=64
fun stuff ;-)

-Yonik
http://www.lucidimagination.com

On Tue, Sep 22, 2009 at 1:42 PM, Michael McCandless
<lucene@mikemccandless.com> wrote:
> Well described, that's exactly it!  I like the concrete example :)
>
> Thanks Yonik.
>
> Mike
>
> On Tue, Sep 22, 2009 at 1:38 PM, Yonik Seeley
> <yonik@lucidimagination.com> wrote:
>> OK Mike, thanks for your patience - I understand now :-)
>>
>> Here's an example that helped me understand - hopefully it will add to
>> others understanding more than it confuses ;-)
>>
>> IW.getReader() => segments={A, B}
>>  // something causes a merge of A,B into AB to start
>> addDoc(doc1)
>>  // doc1 goes into segment C
>> IW.getReader() => segments={A, B, C}
>>  // merge isn't done yet, so getReader() still returns A,B instead of
>> AB, but doc1 is still searchable!
>>
>> OK, in this scenario, there's no advantage to warming in the IW vs the app.
>> Let's start over with a little different timing:
>>
>> segments={A,B}
>>  // something causes a merge of A,B into AB to start
>> addDoc(doc1)
>>  // doc1 goes into segment C
>>  // merging of A,B into AB finishes
>> IW.getReader() => segments={AB, C}
>>
>> Oh, no... with warming at the app level, we need to warm the huge AB
>> segment before doc1 is visible.  We could continue using the old
>> reader while the warming is ongoing, so no user requests will
>> experience long queries, but doc1 isn't in the old segment.
>>
>> With warming in the IW (basically warming becomes part of the same
>> operation as merging), then getReader() would return segments={A,B,C}
>> and doc1 would still be instantly searchable.
>>
>> The only way to duplicate this functionality at the app layer would be
>> to recognize that there is a new segment, try and figure out what old
>> segments were merged to create this new segment, and create a reader
>> that's a mix of old and new to avoid unwarmed segments - not nice.
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message