lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Beady Geraghty <beadygerag...@gmail.com>
Subject Re: standardTokenizer - how to terminate at End of Stream
Date Wed, 21 Sep 2005 16:17:16 GMT
I see some definitions in StandardTokenizerConstants.java
Perhaps these are the values for t.kind.
 Perhaps, I was confused between between the usage of
getNextToken() and next() in the standard analyzer.
When should one use getNextToken() instead of next()
 I am just starting to use Lucene, so please excuse these
simple questions.
 Thanks


 On 9/21/05, Beady Geraghty <beadygeraghty@gmail.com> wrote:
>
> Could someone tell me how to use the StandardTokenizer properly ?
>  I thought that if the tokenizer.getNextToken() returns null, then it is
> the end of stream. I have a loop that tries to get the next token until
> it is null. But the loop doesn't terminate.
> I tried to termintae the loop by t.kind == 0, and it seems to have stopped
> upon the end of stream. I am not sure what t.kind really is.
> The code mentioned that it is defined in Constants.java, and I looked
> that up, but it is apparent not the right file. Maybe I am pointing to
> a wrong directory.
>
>
> StandardTokenizer tokenizer = new StandardTokenizer( r ); // r is a reader
> int count = 0;
> Token t = tokenizer .getNextToken();
>
> while (t != null) {
> count++;
> //if (t.kind == 0)
> // break;
> System.out.println( t );
> t = tokenizer.getNextToken() ;
> System.out.println(count);
> }
> System.out.println ( "done");
>  Thank you for any input.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message