lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: How to leverage the LogMergePolicy "calibrateSizeByDeletes" patch in Solr ?
Date Tue, 22 Sep 2009 17:42:35 GMT
Well described, that's exactly it!  I like the concrete example :)

Thanks Yonik.

Mike

On Tue, Sep 22, 2009 at 1:38 PM, Yonik Seeley
<yonik@lucidimagination.com> wrote:
> OK Mike, thanks for your patience - I understand now :-)
>
> Here's an example that helped me understand - hopefully it will add to
> others understanding more than it confuses ;-)
>
> IW.getReader() => segments={A, B}
>  // something causes a merge of A,B into AB to start
> addDoc(doc1)
>  // doc1 goes into segment C
> IW.getReader() => segments={A, B, C}
>  // merge isn't done yet, so getReader() still returns A,B instead of
> AB, but doc1 is still searchable!
>
> OK, in this scenario, there's no advantage to warming in the IW vs the app.
> Let's start over with a little different timing:
>
> segments={A,B}
>  // something causes a merge of A,B into AB to start
> addDoc(doc1)
>  // doc1 goes into segment C
>  // merging of A,B into AB finishes
> IW.getReader() => segments={AB, C}
>
> Oh, no... with warming at the app level, we need to warm the huge AB
> segment before doc1 is visible.  We could continue using the old
> reader while the warming is ongoing, so no user requests will
> experience long queries, but doc1 isn't in the old segment.
>
> With warming in the IW (basically warming becomes part of the same
> operation as merging), then getReader() would return segments={A,B,C}
> and doc1 would still be instantly searchable.
>
> The only way to duplicate this functionality at the app layer would be
> to recognize that there is a new segment, try and figure out what old
> segments were merged to create this new segment, and create a reader
> that's a mix of old and new to avoid unwarmed segments - not nice.
>
> -Yonik
> http://www.lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message