Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (asf.osuosl.org: domain of TAigner@wescodist.com designates
 12.29.179.202 as permitted sender)
Content-Class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: Optimization
Date: Fri, 7 Oct 2005 09:17:04 -0400
Message-ID: <14FBF41EF1411B45B2EC4ADEAC53D131040BFA2F@MAIL01.wescodist.com>
Thread-Topic: Optimization
Thread-Index: AcXJsLBozsivTsxvSbCWOTja7/JUKwBj5cQg
From: "Aigner, Thomas" <TAigner@WescoDist.com>
To: <java-user@lucene.apache.org>

Thanks Erik, I tried the reverse index and it worked like a charm.
While I was doing this, we figured out a way to handle contains within
search and wildcard searches at the beginning.  I thought I would share
it with the community (and realized it handled the reverse index as
well)

Word: ABCDEFG

Tokens created:
	<ABCDEFG
	BCDEFG
	CDEFG
	DEFG
	EFG
	FG
=09
What I do is if the search string is :=20
	WORD*   I search for <WORD* =20
	*WORD   I search for WORD*
	*WORD*  I search for WORD*
      WORD    I search for <WORD

With this technique, the search result time was decreased tremendously
for contains within and wildcard searches from the beginning.  The index
has become 5X as large and takes longer to build, but I'm willing to
sacrifice disk space and time for this huge benefit of speed.  Also, I
have taken the wildcard query completely out of the program now so
everything uses my customized analyzer.

Tom

-----Original Message-----
From: Erik Hatcher [mailto:erik@ehatchersolutions.com]=20
Sent: Wednesday, October 05, 2005 9:27 AM
To: java-user@lucene.apache.org
Subject: Re: Optimization


On Oct 5, 2005, at 9:05 AM, Aigner, Thomas wrote:
>     Have a question.. Is there any obvious things that can be done
> to help speed up query lookups especially wildcard searches (i.e.
> *lamps).

Obvious?  Sort of.  *lamps needs to scan through _every_ single term =20
in the index (for the specified field only, of course) because terms =20
are lexicographically ordered.

If you reverse terms during analysis and lay them in the same =20
position (increment 0) as the original token you'd end up with =20
"spmal..." terms.  Now pre-process the query string and if there is a =20
prefixed wildcard query, reverse it so that "*lamps" turns into =20
"spmal*" and you will likely achieve a dramatic speed-up.

This is just one technique for dealing with prefixed wildcard =20
queries.  There is more fun to be had with queries like *lamps*.  A =20
technique I learned from the book Managing Gigabytes is to rotate =20
terms through all their possible variations and index all of those, =20
which also requires cleverness on the querying side of things.

     Erik


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org