lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From DM Smith <dmsmith...@gmail.com>
Subject Re: TokenStream and Token APIs
Date Mon, 13 Oct 2008 23:16:11 GMT

On Oct 13, 2008, at 3:34 PM, Doug Cutting wrote:

> Michael Busch wrote:
>>    public abstract boolean nextToken() throws IOException;
>
> What's the point of a separate Token and TokenStream if there's only  
> a single Token per TokenStream?  If that's really the direction  
> we'll go, then all of the Token methods should be on TokenStream,  
> and Token should disappear.  Are there cases where a stream might  
> switch token classes midstream?  If not, then a single, combined API  
> should suffice.

There are several streams that analyze the input and output several  
tokens for each one in the stream. For example, synonyms, shingles,  
stemming.

There are also some caching TokenStreams that can be reset to replay  
their stream.


-- DM


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message