lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <>
Subject RE: [jira] Commented: (LUCENE-1794) implement reusableTokenStream for all contrib analyzers
Date Tue, 11 Aug 2009 13:38:34 GMT
> Just as note related to this discussion:
> TokenFilter#reset says:
>   /** Reset the filter as well as the input TokenStream. */
> However, CachingTokenFilter does not reset the input TokenStream.

That's a bug :-) but it is not a problem, as CachingTokenFilter will not
call the input filter again, when it was reset (it then only delivers cached

TokenFilters should really do a "reset" on reset() and should have a
rewind() method (interface "Rewindable") if they cache Tokens. The problem
is BW compatibility for CachingTokenFilter.

The new TeeSinkTokenFilter is new and can conform to this spec. Deprecated
Tee/SinkTokenizers have the same problem like CachingTokenFilter.


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message