From java-dev-return-14882-apmail-lucene-java-dev-archive=lucene.apache.org@lucene.apache.org Sat Jul 08 01:35:09 2006 Return-Path: Delivered-To: apmail-lucene-java-dev-archive@www.apache.org Received: (qmail 68915 invoked from network); 8 Jul 2006 01:35:08 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199) by minotaur.apache.org with SMTP; 8 Jul 2006 01:35:08 -0000 Received: (qmail 1171 invoked by uid 500); 8 Jul 2006 01:35:06 -0000 Delivered-To: apmail-lucene-java-dev-archive@lucene.apache.org Received: (qmail 1136 invoked by uid 500); 8 Jul 2006 01:35:06 -0000 Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-dev@lucene.apache.org Delivered-To: mailing list java-dev@lucene.apache.org Received: (qmail 1120 invoked by uid 99); 8 Jul 2006 01:35:06 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (140.211.166.49) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Jul 2006 18:35:06 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: neutral (asf.osuosl.org: local policy) Received: from [209.86.89.66] (HELO elasmtp-spurfowl.atl.sa.earthlink.net) (209.86.89.66) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Jul 2006 18:35:05 -0700 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=dk20050327; d=ix.netcom.com; b=k4w77cquL2W1te+4K/ZP2XPcJLUbanyZLb/Ru0c5VWvR4gj6nyzSujy1QYFx4O+1; h=Received:Mime-Version:In-Reply-To:References:Content-Type:Message-Id:Content-Transfer-Encoding:From:Subject:Date:To:X-Mailer:X-ELNK-Trace:X-Originating-IP; Received: from [66.245.135.50] (helo=[192.168.1.119]) by elasmtp-spurfowl.atl.sa.earthlink.net with asmtp (TLSv1:RC4-SHA:128) (Exim 4.34) id 1Fz1iW-0004F4-AU for java-dev@lucene.apache.org; Fri, 07 Jul 2006 21:34:44 -0400 Mime-Version: 1.0 (Apple Message framework v752.2) In-Reply-To: References: Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <6B67DAF0-CFC5-4ED9-88C8-07711C13D2C3@ix.netcom.com> Content-Transfer-Encoding: 7bit From: robert engels Subject: Re: MultiSegmentQueryFilter enhancement for interactive indexes? Date: Fri, 7 Jul 2006 20:34:44 -0500 To: java-dev@lucene.apache.org X-Mailer: Apple Mail (2.752.2) X-ELNK-Trace: 33cbdd8ed9881ca8776432462e451d7bd15d05d9470ff71080d0487a1c89a485da21014df4555f29350badd9bab72f9c350badd9bab72f9c350badd9bab72f9c X-Originating-IP: 66.245.135.50 X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N I implemented it and it works great. I didn't worry about the deletions since by the time a filter is used the deleted documents are already removed by the query. The only problem that arose out of this was for things like the ConstantScoreQuery (which uses a filter) - I needed to modify this query to ignore deleted documents. Now I have incremental cached filters - the query performance is going through the roof. On Jul 7, 2006, at 2:47 PM, Chris Hostetter wrote: > > I'm no segments/MultiReader expert, but your idea sounds good to > me ... it > seems like it would certainly work in the "new segments" situation. > > One thing i don't see you mention is dealing with deletions ... i'm > not > sure if deleting documents cause the version number of an > IndexReader to > change or not (if it does your job is easy) but even if it doesn't I'm > guessing you could say that if hasDeletions() returns true, you > have to > assume you need to invalidate your cached bits (worst case scenerio > you > are invalidating the cache as often as it is now) > > > : Date: Fri, 7 Jul 2006 00:32:54 -0500 > : From: robert engels > : Reply-To: java-dev@lucene.apache.org > : To: Lucene-Dev > : Subject: MultiSegmentQueryFilter enhancement for interactive > indexes? > : > : I thought of a possible enhancement - before I go down the road, > I am > : looking for some input form the community? > : > : Currently, the QueryFilter caches the bits base upon the > IndexReader. > : > : The problem with this is small incremental changes to the index > : invalidate the cache. > : > : What if instead the filter determined that the underlying > IndexReader > : was a MultiReader and then maintained a bitset for each reader, > : combining them in bits() when requested. The filter could check if > : any of the underlying readers were the different (removed or added) > : and then just create a new bitset for that reader. With the new non- > : bit set filter implementations this could be even more memory > : efficient since the bitsets would not need to be combined into a > : single bitset. > : > : With the previous work on "reopen" so that segments are reused, this > : would allow filters to be far more useful in a highly interactive > : environment. > : > : What do you think? > : > : > : > : > : > : > : > --------------------------------------------------------------------- > : To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org > : For additional commands, e-mail: java-dev-help@lucene.apache.org > : > > > > -Hoss > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org > For additional commands, e-mail: java-dev-help@lucene.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org For additional commands, e-mail: java-dev-help@lucene.apache.org