Mailing-List: contact general-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@lucene.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Content-Class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Subject: RE: Simplest way to check for an exact match on an tokenized/stored
 field?
Date: Mon, 27 Oct 2008 13:18:59 -0400
Message-ID: <9294E20AED46934EA459020706463F94EF40A5@SUEXCL-02.ad.syr.edu>
In-Reply-To: <20178452.post@talk.nabble.com>
Thread-Topic: Simplest way to check for an exact match on an tokenized/stored
 field?
Thread-Index: Ack3t5a21X3L2Cw9RY6GgE6+kNYwFwAnzfCw
From: Steven A Rowe <sarowe@syr.edu>
To: <general@lucene.apache.org>

Hi chaiguy1337,

On 10/26/2008 at 6:09 PM, chaiguy1337 wrote:
> Hi group. I have a Lucene index that contains a bunch of text =
documents,
> which are both tokenized (using the standard analyzer, not
> KeywordAnalyzer) and stored. Preferrably without having to create a
> duplicate KeywordAnalyzer-tokenized field, what is the simplest =
(and/or
> most efficient) way to check for an existing exact match on that =
field?
>=20
> Currently my best guess is to perform a TermQuery containing
> the entire text of the document to check, and then perform a
> second pass over each of the results checking the field for
> explicit equality.

The StandardAnalyzer can produce the same set of tokens for two =
non-identical texts, especially if you are using stop words, so =
depending on how strictly you define "exact match", you may have to =
re-index.

What are you trying to do?  If you're searching for duplicates, it may =
make sense for you to compute a digest of some form and store that for =
comparison purposes in another field.

Steve