lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Hostetter <>
Subject Re: standardTokenizer - how to terminate at End of Stream
Date Thu, 22 Sep 2005 03:43:35 GMT

:  Since I used the StandAnalyzer when I originally created the index,
: I therefore use the StandardTokenizer to tokenize the input stream.
:  Is there a better way to do what I try to do ?
:   From your comment below, it appears that I should just use next() instead

if your goal is to recreate the tokens you get from using
StandardAnalyzer, then don't use StandardTokenizer -- use
StandardAnalyzer.  It does other things besides tokenizing.  get the
TokenStream from StandardAnalyzer and it's next() method should do what
you want.

None of which should imply that this is the best way to achieve your goal
-- i'm sure the highlighter package will do what you want, but i've never
used it personally.


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message