lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yonik Seeley <yo...@lucidimagination.com>
Subject Re: How to leverage the LogMergePolicy "calibrateSizeByDeletes" patch in Solr ?
Date Tue, 22 Sep 2009 17:38:12 GMT
OK Mike, thanks for your patience - I understand now :-)

Here's an example that helped me understand - hopefully it will add to
others understanding more than it confuses ;-)

IW.getReader() => segments={A, B}
  // something causes a merge of A,B into AB to start
addDoc(doc1)
  // doc1 goes into segment C
IW.getReader() => segments={A, B, C}
  // merge isn't done yet, so getReader() still returns A,B instead of
AB, but doc1 is still searchable!

OK, in this scenario, there's no advantage to warming in the IW vs the app.
Let's start over with a little different timing:

segments={A,B}
  // something causes a merge of A,B into AB to start
addDoc(doc1)
  // doc1 goes into segment C
  // merging of A,B into AB finishes
IW.getReader() => segments={AB, C}

Oh, no... with warming at the app level, we need to warm the huge AB
segment before doc1 is visible.  We could continue using the old
reader while the warming is ongoing, so no user requests will
experience long queries, but doc1 isn't in the old segment.

With warming in the IW (basically warming becomes part of the same
operation as merging), then getReader() would return segments={A,B,C}
and doc1 would still be instantly searchable.

The only way to duplicate this functionality at the app layer would be
to recognize that there is a new segment, try and figure out what old
segments were merged to create this new segment, and create a reader
that's a mix of old and new to avoid unwarmed segments - not nice.

-Yonik
http://www.lucidimagination.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message