Return-Path: Delivered-To: apmail-lucene-java-dev-archive@www.apache.org Received: (qmail 10009 invoked from network); 23 Sep 2009 09:13:19 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 23 Sep 2009 09:13:19 -0000 Received: (qmail 64241 invoked by uid 500); 23 Sep 2009 09:13:18 -0000 Delivered-To: apmail-lucene-java-dev-archive@lucene.apache.org Received: (qmail 64165 invoked by uid 500); 23 Sep 2009 09:13:18 -0000 Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-dev@lucene.apache.org Delivered-To: mailing list java-dev@lucene.apache.org Received: (qmail 64157 invoked by uid 99); 23 Sep 2009 09:13:18 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Sep 2009 09:13:18 +0000 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.85.211.188] (HELO mail-yw0-f188.google.com) (209.85.211.188) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Sep 2009 09:13:07 +0000 Received: by ywh26 with SMTP id 26so385881ywh.29 for ; Wed, 23 Sep 2009 02:12:45 -0700 (PDT) MIME-Version: 1.0 Received: by 10.150.63.13 with SMTP id l13mr3846711yba.5.1253697165489; Wed, 23 Sep 2009 02:12:45 -0700 (PDT) In-Reply-To: <4AB96FDB.8090408@gmail.com> References: <8837fb770909212156s13645f84tbee15ee3bd2d5aef@mail.gmail.com> <8837fb770909221602u6b9cd5ccqf3434c1c542bb1d0@mail.gmail.com> <4AB95A7D.7080601@gmail.com> <9ac0c6aa0909221617v333482em36993ec9f642cdfc@mail.gmail.com> <8837fb770909221643h1ef1a23v34e6655b64d92313@mail.gmail.com> <4AB963F5.6040107@gmail.com> <8837fb770909221726h4822f8ceo527ad0905b7e9183@mail.gmail.com> <4AB96CF4.4080101@gmail.com> <8837fb770909221739h27dcdb91i4d1de1d282eb8149@mail.gmail.com> <4AB96FDB.8090408@gmail.com> Date: Wed, 23 Sep 2009 05:12:45 -0400 Message-ID: <9ac0c6aa0909230212m10694abfqd67bce35f2a81cd9@mail.gmail.com> Subject: Re: 2.9 NRT w.r.t. sorting and field cache From: Michael McCandless To: java-dev@lucene.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org The cat is awake! And catching up on all these interesting emails... I think Mark said it all very well :) The warming you actually do in what you pass to setMergedSegmentWarmer, I think, should look just like the "normal" warming you'd do before bringing a newly opened reader into production. Lucene must load norms per searched field, so run one search per such field. It loads FieldCache entry for sorting/function queries, so run one search for each such field. As of 2.9, terms dict index is now loaded when you open the reader. Then, if you want to warm the OS's IO cache to match your "frequent queries" you need to carry over some sort cache that tracks such queries (like Solr). The warming is the same, but when you do it in IndexWriter, when a large segment merge completes, it won't block your ongoing streams of updates which is usually crucial in a large scale NRT app. Nevertheless, the shear CPU and IO cost of merging, and Java's inability to down-prioritize merging IO and control the OS's IO cache, will still impact search performance. So it may still be necessary to entirely avoid large segment merges. Mike On Tue, Sep 22, 2009 at 8:46 PM, Mark Miller wrote: > What I would do is: > > In the warm method, load a FieldCache for every field I was going to end > up using a FieldCache for. > If its just for sorting, I might do a search with a sort on every field > I was going to sort on. > That will get the segment FieldCaches into RAM before the SegmentReader > is put into use. > > I might also do a search or two that hits a lot of terms to get some of > the index into RAM. Or maybe walk a termenum- or anything one normally > does when warming Readers (like Solr does, or many other home grown > solutions.) > > I don't think there is anything special in this case. You don't have to > hit it with every unique search you expect it to see - you just get some > key pieces (especially the FieldCaches) into RAM. > > Don't give mike a hard time about his valuable time - I'm sure he would > have answered, but he's likely in bed (That cat wakes early it seems. ). > He's a lot nicer than I am ;) > > John Wang wrote: >> No worries. >> Just trying to understand things. >> >> I wanted to double check but didn't want to write "My IDE told me that >> was the case" to sound pissy. >> >> I did look at the code, sometimes too much actually, but I never want >> to claim I understand the code 100%, hence going to the source is >> probably the best, even at the expense of sounding dumb, it is usually >> worthy it ;) >> >> My question is more on how would a person do it on the public API >> level without having to hack into the source code. >> >> My main misunderstanding at this point is that I had thought >> IndexReaderWarmer can directly warm the field cache deterministically. >> >> Thanks >> >> -John >> >> On Wed, Sep 23, 2009 at 8:33 AM, Mark Miller > > wrote: >> >> =A0 =A0 Don't take me too seriously John - I doubt anyone does :) >> >> =A0 =A0 And I wasn't implying Mike's time was more valuable than yours. = I was >> =A0 =A0 being ... uh ... me :) >> >> =A0 =A0 And I don't claim that all of your many questions could have bee= n >> =A0 =A0 found >> =A0 =A0 in 5 seconds ;) >> >> =A0 =A0 Just the ones you were asking - its very quick (at least with ec= lipse) >> =A0 =A0 to see that there is no default impl. >> =A0 =A0 Its also very quick to see that a segment reader is passed to th= e warm >> =A0 =A0 method every time. I think its just >> =A0 =A0 a generic IndexReader because you would warm a multi-reader the >> =A0 =A0 same way >> =A0 =A0 as a segmentreader. >> >> =A0 =A0 I was just suggesting you look at the code a bit, because I thin= k its >> =A0 =A0 fairly easy to figure out the basics of the warmer (hey, if I ca= n >> =A0 =A0 do it >> =A0 =A0 ;) ). >> >> =A0 =A0 Again, don't take me too seriously. I send out my comments faste= r >> =A0 =A0 than I >> =A0 =A0 can think of them. And I've probably wasted more of Mike's time >> =A0 =A0 than anyone. >> >> =A0 =A0 The only way you will load the entire FieldCache is to use a top= level >> =A0 =A0 Reader outside of the core API - the core api works per segment >> =A0 =A0 now. And >> =A0 =A0 the IndexReaderWarmer is always passed a segmentreader from the >> =A0 =A0 readerPool. >> >> =A0 =A0 - Mark >> >> =A0 =A0 John Wang wrote: >> =A0 =A0 > Mark: >> =A0 =A0 > >> =A0 =A0 > I did spend at least a quarter of an ounce. :) And I am sure M= ike's >> =A0 =A0 > time is more valuable than mine, but it was meant to be a >> =A0 =A0 "double-check" >> =A0 =A0 > >> =A0 =A0 > I was under the impression there is a default impl from previo= us >> =A0 =A0 email >> =A0 =A0 > threads on how to handle field cache warming, perhaps I >> =A0 =A0 misunderstood. >> =A0 =A0 > >> =A0 =A0 > The real question here is "warms the reader" From a public API= point >> =A0 =A0 > of view, I wasn't sure if passing in a IndexReader impl is som= ething >> =A0 =A0 > we can do to avoid loading the entire field cache. e.g. would = I need >> =A0 =A0 > to down cast? can it be a filtered reader? etc. >> =A0 =A0 > >> =A0 =A0 > If you think there is something I could have done witin 5 sec, >> =A0 =A0 please >> =A0 =A0 > point me to the right direction. >> =A0 =A0 > >> =A0 =A0 > Thanks >> =A0 =A0 > >> =A0 =A0 > -John >> =A0 =A0 > >> =A0 =A0 > On Wed, Sep 23, 2009 at 7:55 AM, Mark Miller >> =A0 =A0 >> =A0 =A0 > >> >> =A0 =A0 wrote: >> =A0 =A0 > >> =A0 =A0 > =A0 =A0 Come on dude :) Spend a half ounce of effort first. Mi= ke's >> =A0 =A0 time is too >> =A0 =A0 > =A0 =A0 valuable ! >> =A0 =A0 > >> =A0 =A0 > =A0 =A0 Luckily mine is not. >> =A0 =A0 > >> =A0 =A0 > =A0 =A0 There is no default impl - the class is dead simple (a= nd the >> =A0 =A0 class has >> =A0 =A0 > =A0 =A0 been pointed out like 3 times in this thread - I'm not= even >> =A0 =A0 fully >> =A0 =A0 > =A0 =A0 following and I know where to find it): >> =A0 =A0 > >> =A0 =A0 > =A0 =A0 =A0public static abstract class IndexReaderWarmer { >> =A0 =A0 > =A0 =A0 =A0 =A0public abstract void warm(IndexReader reader) t= hrows >> =A0 =A0 IOException; >> =A0 =A0 > =A0 =A0 =A0} >> =A0 =A0 > >> =A0 =A0 > =A0 =A0 Now pass something in that warms the reader. Load a >> =A0 =A0 fieldcache - do a >> =A0 =A0 > =A0 =A0 search. Do the hokey pokey and turn your self around .= .. >> =A0 =A0 > >> =A0 =A0 > =A0 =A0 Investigation time: 5 seconds. >> =A0 =A0 > >> =A0 =A0 > =A0 =A0 John Wang wrote: >> =A0 =A0 > =A0 =A0 > Hi Michael: >> =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 =A0Thanks for the pointer! >> =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 =A0 Pardon my ignorance, but I am still no s= eeing the >> =A0 =A0 connection >> =A0 =A0 > =A0 =A0 > between this api to per/segment loading of FieldCach= e. >> =A0 =A0 (the api >> =A0 =A0 > =A0 =A0 takes >> =A0 =A0 > =A0 =A0 > in an IndexReader instead of maybe SegmentReader[]) >> =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 =A0 Can you point me to maybe the default im= pl of >> =A0 =A0 > =A0 =A0 IndexReaderWarmer >> =A0 =A0 > =A0 =A0 > to help me understand? >> =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > Thanks >> =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > -John >> =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > On Wed, Sep 23, 2009 at 7:17 AM, Michael McCandless >> =A0 =A0 > =A0 =A0 > > =A0 =A0 >> =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> wrote: >> =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 This is exactly why we added >> =A0 =A0 > =A0 =A0 IndexWriter.setMergedSegmentWarmer -- you >> =A0 =A0 > =A0 =A0 > =A0 =A0 can warm the reader w/o blocking ongoing upd= ates. >> =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 Mike >> =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 On Tue, Sep 22, 2009 at 7:15 PM, Mark Miller >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 > >> =A0 =A0 > =A0 =A0 >> =A0 =A0 >>> = wrote: >> =A0 =A0 > =A0 =A0 > =A0 =A0 > Right - when a large segment is invalidate= d, you >> =A0 =A0 will have >> =A0 =A0 > =A0 =A0 a bigger >> =A0 =A0 > =A0 =A0 > =A0 =A0 > fieldcache piece to reload - pre 2.9, you'= d be reloading >> =A0 =A0 > =A0 =A0 the *whole* >> =A0 =A0 > =A0 =A0 > =A0 =A0 > field cache every time though. Sounds like= you are >> =A0 =A0 trying to >> =A0 =A0 > =A0 =A0 > =A0 =A0 deal with >> =A0 =A0 > =A0 =A0 > =A0 =A0 > those large segments changing anyway :) Th= ey are >> =A0 =A0 always an >> =A0 =A0 > =A0 =A0 issue >> =A0 =A0 > =A0 =A0 > =A0 =A0 when >> =A0 =A0 > =A0 =A0 > =A0 =A0 > doing RT it seems. >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 > I don't believe deletes invalidate a field= cache - >> =A0 =A0 terms from >> =A0 =A0 > =A0 =A0 > =A0 =A0 deleted >> =A0 =A0 > =A0 =A0 > =A0 =A0 > docs stay in a field cache and segmentread= ers use their >> =A0 =A0 > =A0 =A0 > =A0 =A0 freqStream as >> =A0 =A0 > =A0 =A0 > =A0 =A0 > the fieldcache key. Only when the deletes = are merged out >> =A0 =A0 > =A0 =A0 would they >> =A0 =A0 > =A0 =A0 > =A0 =A0 > invalidate - but because your writing a ne= w segment >> =A0 =A0 anyway ... >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 > - Mark >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 > John Wang wrote: >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> I understand what you are saying. Let me = detail >> =A0 =A0 what I am >> =A0 =A0 > =A0 =A0 > =A0 =A0 trying to say: >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> When "currently processed segments" are f= lushed down, >> =A0 =A0 > =A0 =A0 merge may >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> happen. When merges happen, some of those= "stable >> =A0 =A0 > =A0 =A0 segments" will be >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> invalidated, and so will the fieldcache d= ata keyed >> =A0 =A0 by them. >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> In a high update environment, such scenar= ios can >> =A0 =A0 happen quite >> =A0 =A0 > =A0 =A0 > =A0 =A0 often. >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> The way the default mergePolicy works is = that small >> =A0 =A0 > =A0 =A0 segments get >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> merged into the larger segments. Eventual= ly, what >> =A0 =A0 will be >> =A0 =A0 > =A0 =A0 > =A0 =A0 invalidated >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> would be a large segment, and when that h= appens, a >> =A0 =A0 large >> =A0 =A0 > =A0 =A0 chunk >> =A0 =A0 > =A0 =A0 > =A0 =A0 of the >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> field cache would be invalidated. >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> Furthermore, in the case where there are = high updates, >> =A0 =A0 > =A0 =A0 the stable >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> segments can be invalidate much sooner wh= en there >> =A0 =A0 are deletes >> =A0 =A0 > =A0 =A0 > =A0 =A0 in those >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> segments, and I would guess the correspon= ding >> =A0 =A0 FieldCache >> =A0 =A0 > =A0 =A0 needs >> =A0 =A0 > =A0 =A0 > =A0 =A0 to be >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> adjusted. Not sure how it is handled righ= t now. >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> Just my two cents, and of course when I f= ind the >> =A0 =A0 time I will >> =A0 =A0 > =A0 =A0 > =A0 =A0 need to >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> run some tests to see. >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> -John >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> On Tue, Sep 22, 2009 at 3:59 PM, Uwe Schi= ndler >> =A0 =A0 > =A0 =A0 >> =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > >> =A0 =A0 > =A0 =A0 >> =A0 =A0 >>>> wrote: >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 The NRT reader coming from the >> =A0 =A0 > =A0 =A0 IndexWriter.getReader() has only >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 changes in the currently processe= d segments, the >> =A0 =A0 > =A0 =A0 other segments >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 keep stable (and even their Index= Reader keys >> =A0 =A0 used for the >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 FieldCache). The rest of the segm= ents keep stable. >> =A0 =A0 > =A0 =A0 For the >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 consumer it looks like a normal r= eader (it is >> =A0 =A0 in fact a >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 ReadOnlyDirectoryReader) supporti= ng >> =A0 =A0 > =A0 =A0 > =A0 =A0 getSequentialSubReaders() and >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 so on. >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 ----- >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 Uwe Schindler >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 H.-H.-Meier-Allee 63, D-28213 Bre= men >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 http://www.thetaphi.de >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 eMail: uwe@thetaphi.de >> =A0 =A0 > >> =A0 =A0 > =A0 =A0 >> =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 > >> =A0 =A0 > =A0 =A0 >> =A0 =A0 >>> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > >> =A0 =A0 > >> =A0 =A0 ----------------------------------------------------------------= -------- >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 *From:* John Wang [mailto:john.wa= ng@gmail.com >> =A0 =A0 >> =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 > =A0 =A0 >> =A0 =A0 > =A0 =A0 > >> =A0 =A0 >> =A0 =A0 > =A0 =A0 >>>] >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 *Sent:* Tuesday, September 22, 20= 09 9:32 AM >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 *To:* java-dev@lucene.apache.org >> =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 *Subject:* Re: 2.9 NRT w.r.t. sor= ting and field >> =A0 =A0 cache >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 Thanks Mark for the pointer! >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 I guess my point is with NRT, and= when segment >> =A0 =A0 files >> =A0 =A0 > =A0 =A0 change >> =A0 =A0 > =A0 =A0 > =A0 =A0 often, >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 this would be an issue, no? >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 Anyway, I can run some tests. >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 Thanks >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 -John >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 On Tue, Sep 22, 2009 at 3:21 PM, = Mark Miller >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 >> =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 >> =A0 =A0 >>>> >> =A0 =A0 wrote: >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 1483 - indexsearcher pulls out a = readers subreaders >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 (segmentreaders) and sends a coll= ector over >> =A0 =A0 them one >> =A0 =A0 > =A0 =A0 by one, >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 rather than using the multireader= . So only fc >> =A0 =A0 for seg >> =A0 =A0 > =A0 =A0 > =A0 =A0 readers that >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 change need to be reloaded. >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 - Mark >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 http://www.lucidimagination.com (= mobile) >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 On Sep 22, 2009, at 1:27 AM, John= Wang >> =A0 =A0 > =A0 =A0 >> =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 > =A0 =A0 >> =A0 =A0 > =A0 =A0 > >> =A0 =A0 >> =A0 =A0 > =A0 =A0 >>>> >> =A0 =A0 > =A0 =A0 > =A0 =A0 wrote: >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 Hi Yonik: >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 =A0 =A0 =A0Actually that is what= I am looking for. >> =A0 =A0 Can you >> =A0 =A0 > =A0 =A0 > =A0 =A0 please point >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 me to where/how sorting is done = per-segment? >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 =A0 =A0 =A0When heaving indexing= introduces or modifies >> =A0 =A0 > =A0 =A0 > =A0 =A0 segments, would >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 it cause reloading of FieldCache= at query time and >> =A0 =A0 > =A0 =A0 thus would >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 impact search performance? >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 thanks >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 -John >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 On Tue, Sep 22, 2009 at 1:05 PM,= Yonik Seeley >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 > =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>>> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 wrote: >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 On Tue, Sep 22, 2009 at 12:56 AM= , John Wang >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 > >> =A0 =A0 > =A0 =A0 >> =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 > =A0 =A0 >> =A0 =A0 > =A0 =A0 > >> =A0 =A0 >> =A0 =A0 > =A0 =A0 >>>> >> =A0 =A0 > =A0 =A0 > =A0 =A0 wrote: >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 > Looking at the code, seems the= re is a disconnect >> =A0 =A0 > =A0 =A0 between >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 how/when field >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 > cache is loaded when IndexWrit= er.getReader() is >> =A0 =A0 > =A0 =A0 called. >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 I'm not sure what you mean by "d= isconnect" >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 > Is FieldCache updated? >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 FieldCache entries are populated= on demand, as >> =A0 =A0 they >> =A0 =A0 > =A0 =A0 always >> =A0 =A0 > =A0 =A0 > =A0 =A0 have been. >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 > Otherwise, are we reloading Fi= eldCache for each >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 > reader instance? >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 Searching/sorting is now per-seg= ment, and so >> =A0 =A0 is the >> =A0 =A0 > =A0 =A0 use of the >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 FieldCache. =A0Segments that don= 't change shouldn't >> =A0 =A0 > =A0 =A0 have to >> =A0 =A0 > =A0 =A0 > =A0 =A0 reload >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 their >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 FieldCache entries. >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 -Yonik >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 http://www.lucidimagination.com >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> >> =A0 =A0 > =A0 =A0 > >> =A0 =A0 > >> =A0 =A0 ----------------------------------------------------------------= ----- >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 To unsubscribe, e-mail: >> =A0 =A0 > =A0 =A0 > =A0 =A0 java-dev-unsubscribe@lucene.apache.org >> =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 > =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 For additional commands, e-mail: >> =A0 =A0 > =A0 =A0 > =A0 =A0 java-dev-help@lucene.apache.org >> =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> =A0 =A0 > =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >>> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 > -- >> =A0 =A0 > =A0 =A0 > =A0 =A0 > - Mark >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 > http://www.lucidimagination.com >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > >> =A0 =A0 > >> =A0 =A0 ----------------------------------------------------------------= ----- >> =A0 =A0 > =A0 =A0 > =A0 =A0 > To unsubscribe, e-mail: >> =A0 =A0 > =A0 =A0 java-dev-unsubscribe@lucene.apache.org >> =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 > For additional commands, e-mail: >> =A0 =A0 > =A0 =A0 java-dev-help@lucene.apache.org >> =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > >> =A0 =A0 > >> =A0 =A0 ----------------------------------------------------------------= ----- >> =A0 =A0 > =A0 =A0 > =A0 =A0 To unsubscribe, e-mail: >> =A0 =A0 > =A0 =A0 java-dev-unsubscribe@lucene.apache.org >> =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > =A0 =A0 For additional commands, e-mail: >> =A0 =A0 > =A0 =A0 java-dev-help@lucene.apache.org >> =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > =A0 =A0 > =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 >> >> =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 > >> =A0 =A0 > >> =A0 =A0 > >> =A0 =A0 > =A0 =A0 -- >> =A0 =A0 > =A0 =A0 - Mark >> =A0 =A0 > >> =A0 =A0 > =A0 =A0 http://www.lucidimagination.com >> =A0 =A0 > >> =A0 =A0 > >> =A0 =A0 > >> =A0 =A0 > >> =A0 =A0 > >> =A0 =A0 ----------------------------------------------------------------= ----- >> =A0 =A0 > =A0 =A0 To unsubscribe, e-mail: >> =A0 =A0 java-dev-unsubscribe@lucene.apache.org >> =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > =A0 =A0 For additional commands, e-mail: >> =A0 =A0 java-dev-help@lucene.apache.org >> =A0 =A0 >> =A0 =A0 > =A0 =A0 > =A0 =A0 > >> =A0 =A0 > >> =A0 =A0 > >> >> >> =A0 =A0 -- >> =A0 =A0 - Mark >> >> =A0 =A0 http://www.lucidimagination.com >> >> >> >> >> =A0 =A0 ----------------------------------------------------------------= ----- >> =A0 =A0 To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org >> =A0 =A0 >> =A0 =A0 For additional commands, e-mail: java-dev-help@lucene.apache.org >> =A0 =A0 >> >> > > > -- > - Mark > > http://www.lucidimagination.com > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org > For additional commands, e-mail: java-dev-help@lucene.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org For additional commands, e-mail: java-dev-help@lucene.apache.org