Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (asf.osuosl.org: domain of Russell.Allen@aebn.net
 designates 216.54.226.11 as permitted sender)
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: PrefixQuery rewrite() bug, ignores max clause count
Date: Mon, 17 Jul 2006 09:18:53 -0400
Message-ID: 
 <E9BF7E0FD95D1143B399B129908A58FF014165E0@datachange01.dataconversions.biz>
In-Reply-To: <Pine.LNX.4.58.0607141607570.3459@hal.rescomp.berkeley.edu>
Thread-Topic: PrefixQuery rewrite() bug, ignores max clause count
Thread-Index: Acanm65mm4y/Z6P6Qr2e7ZI9o0Te7gCBUZQA
From: "Russell M. Allen" <Russell.Allen@aebn.net>
To: <java-user@lucene.apache.org>

: 1) colorzing your mail doesn't play nicely with the mailing list ...
: sending "diffs" is the prefered way to show potential changes to the
code.

So I discovered on sending the message.  :)

: 2) assuming i understand what it is you've changed, you've "worked
arround" the TooManyClausesException in a somewhat complicated way (you
could just as : easily increase the maxClauseCount and save yourslef
some
: headache) ...=20

Actually, we have increased the maxClauseCount in our production code to
address this problem.  (We don't want to maintain a custom version of
Lucene.)

: by doing this you bypass the whole point of having a
: maClauseCount: to prevent a query that will consume too many resources
(namely RAM and query execution time) from being constructed.

I understand the concern for resources, but by throwing an exception you
have also preempted a legitimate query.  How then should one go about
performing a query for all document where a field begins with char X?
We have 60,000+ documents, of which B, S and T seem to be the most
common initial char (just over 7000 document each).  We are building an
'index' with the search results, thus the need for a query like:
"name:b*"

IMHO, both your concern for resources and the ability to query for
"name:b*" are legitimate.  Perhaps PrefixeQuery should be able to
gracefully degrade when it hits resource limits.  I admit I am at the
limit of my Lucene knowledge here, but I assume that rewrite() is an
optimization of the query and can be skipped (with some additional code)
at the cost of performance.=20

: 3) your change modifies the core impacts from each of the Terms that
match the prefix ... if you don't care about the score impacts from the
terms, then : there are other options.  Solr has a PrefixFilter class
which can be wrapped in a ConstantScoreQuery to support prefix queries
regardless of your term : : distribution.

We do not rank or score results at all (Lucene is a high speed index for
us).  As a result, I am blissfully ignorant of scoring results.  Out of
curiosity though, does the depth of a term in a query tree affect its
score?

I will take a look at PrefixFilter and ConstantScoreQuery.

Thanks!
Russell

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org