lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Beady Geraghty <>
Subject standardTokenizer - how to terminate at End of Stream
Date Wed, 21 Sep 2005 15:02:28 GMT
Could someone tell me how to use the StandardTokenizer properly ?
 I thought that if the tokenizer.getNextToken() returns null, then it is
the end of stream. I have a loop that tries to get the next token until
it is null. But the loop doesn't terminate.
I tried to termintae the loop by t.kind == 0, and it seems to have stopped
upon the end of stream. I am not sure what t.kind really is.
The code mentioned that it is defined in, and I looked
that up, but it is apparent not the right file. Maybe I am pointing to
a wrong directory.

StandardTokenizer tokenizer = new StandardTokenizer( r ); // r is a reader
int count = 0;
Token t = tokenizer .getNextToken();

while (t != null) {
//if (t.kind == 0)
// break;
System.out.println( t );
t = tokenizer.getNextToken() ;
System.out.println( "done");
 Thank you for any input.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message