Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Message-Id: <2C396FF4-74E1-4E62-8408-36C4C9DA8DAA@theronge.com>
From: Matt Ronge <mronge@theronge.com>
To: java-user@lucene.apache.org
In-Reply-To: <200808301313.20821.paul.elschot@xs4all.nl>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v926)
Subject: Re: Pre-filtering for expensive query
Date: Sat, 30 Aug 2008 11:22:50 -0500
References: <4EC81374-F485-4A2D-A178-8838235AE6C1@theronge.com>
 <200808301313.20821.paul.elschot@xs4all.nl>


On Aug 30, 2008, at 6:13 AM, Paul Elschot wrote:

> Op Saturday 30 August 2008 03:34:01 schreef Matt Ronge:
>> Hi all,
>>
>> I am working on implementing a new Query, Weight and Scorer that is
>> expensive to run. I'd like to limit the number of documents I run
>> this query on by first building a candidate set of documents with a
>> boolean query. Once I have that candidate set, I was hoping I could
>> build a filter off of it, and issue that along with my expensive
>> query. However, after reading the code I see that filtering is done
>> during the search, and not before hand.
>
> Correct. I suppose you mean the filtering code in IndexSearcher?

Yes, that's exactly what I mean.

>
>> So my initial boolean query
>> won't help in limiting the number of documents scored by my expensive
>> query.
>
> The trick of filtering is the use of skipTo() on both the filter and
> the scorer to skip superfluous work as much as possible.
> So when you make your scorer implement skipTo() efficiently,
> filtering it should reduce the amount of scoring done.
>
> Implementing skipTo() efficiently is normally done by using
> TermScorer.skipTo() on the leafs of a scorer structure. So,
> in case you implement your own TermScorer, take a serious
> look at TermScorer.skipTo().
>
> Normally, score value computations are not the bottleneck,
> but accessing the index is, and this is where skipTo() does
> the real work. At the moment avoiding score value computations
> is a nice extra.

I was not aware of this. Where can I find the code that uses the  
filter to determine what values to feed to skipTo (I'm trying to get a  
better understand of the Lucene source)?

>
>
>> Or should I just implement something myself in a custom scorer?
>
> In case you have a better way than skipTo(), or something
> to improve on this issue to allow a Filter as clause to BooleanQuery:
> https://issues.apache.org/jira/browse/LUCENE-1345
> let us know.


Thanks, if the skipTo approach doesn't work, I'll take a look at this.

--
Matt

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org