Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (asf.osuosl.org: domain of dcliman@keepmedia.com designates
 63.82.1.162 as permitted sender)
From: "Dan Climan" <dcliman@keepmedia.com>
To: <java-user@lucene.apache.org>
Subject: Highlighter, Term Positions and Stopwords
Date: Mon, 5 Dec 2005 20:32:12 -0800
Organization: KeepMedia, Inc.
Message-ID: <003901c5fa1e$06f3c8d0$a701a8c0@corp.keepmedia.int>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_003A_01C5F9DA.F8D088D0"
Importance: Normal

------=_NextPart_000_003A_01C5F9DA.F8D088D0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Do stopfilters create non-contiguous token positions?
=20
I was interested in experimenting with the highlighter and using the
TokenSources.getTokenStream(TermPositionVector
<file:///C:\mysvn\lucene\build\docs\api\org\apache\lucene\index\TermPosit=
ion
Vector.html>  tpv,                                       boolean
tokenPositionsGuaranteedContiguous) method
=20
The javadocs for this method note that:

tokenPositionsGuaranteedContiguous - true if the token position numbers =
have
no overlaps or gaps.

=20

The example used for comparison to re-Analyzing the the text includes
stopwords ("timings above were using a stemmer/lowercaser/stopword =
combo").

I was curious if a stopwords, by definition meant that tokens were not
contiguous? Is this still true if the the query uses the same analyzer =
and
filters out the same stopwords?

=20

Thanks,

Dan


------=_NextPart_000_003A_01C5F9DA.F8D088D0--