lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erik Hatcher <e...@ehatchersolutions.com>
Subject Re: standardTokenizer - how to terminate at End of Stream
Date Wed, 21 Sep 2005 17:56:35 GMT
Could you elaborate on what you're trying to do, please?

Using StandardTokenizer in this low-level fashion is practically  
unheard of, so I think knowing what you're attempting to do will help  
us help you :)

     Erik


On Sep 21, 2005, at 12:17 PM, Beady Geraghty wrote:

> I see some definitions in StandardTokenizerConstants.java
> Perhaps these are the values for t.kind.
>  Perhaps, I was confused between between the usage of
> getNextToken() and next() in the standard analyzer.
> When should one use getNextToken() instead of next()
>  I am just starting to use Lucene, so please excuse these
> simple questions.
>  Thanks
>
>
>  On 9/21/05, Beady Geraghty <beadygeraghty@gmail.com> wrote:
>
>>
>> Could someone tell me how to use the StandardTokenizer properly ?
>>  I thought that if the tokenizer.getNextToken() returns null, then  
>> it is
>> the end of stream. I have a loop that tries to get the next token  
>> until
>> it is null. But the loop doesn't terminate.
>> I tried to termintae the loop by t.kind == 0, and it seems to have  
>> stopped
>> upon the end of stream. I am not sure what t.kind really is.
>> The code mentioned that it is defined in Constants.java, and I looked
>> that up, but it is apparent not the right file. Maybe I am  
>> pointing to
>> a wrong directory.
>>
>>
>> StandardTokenizer tokenizer = new StandardTokenizer( r ); // r is  
>> a reader
>> int count = 0;
>> Token t = tokenizer .getNextToken();
>>
>> while (t != null) {
>> count++;
>> //if (t.kind == 0)
>> // break;
>> System.out.println( t );
>> t = tokenizer.getNextToken() ;
>> System.out.println(count);
>> }
>> System.out.println ( "done");
>>  Thank you for any input.
>>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message