lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler (JIRA)" <>
Subject [jira] Commented: (LUCENE-1804) Can't specify AttributeSource for Tokenizer
Date Wed, 12 Aug 2009 16:35:15 GMT


Uwe Schindler commented on LUCENE-1804:

Normally it would be ok. E.g. in the reuse of TokenStreams, the simpliest would be to create
the tokenizer with a null Reader first and only reset(Reader) it before first use. I think,
this has historical reasons and to keep consistent we should add the ctors. Or deprecate all
Reader ctors and state, that you should create a reusable Tokenizer and call reset(Reader).

I am still not sure, why a simple TokenFilter does not serve the same pupose you would like
to have with Tokenizer here. Why not simply wrap the Tokenizer with a TokenFilter that already
has the possibility to delegate? If it is because you miss the reset(Reader) call, we could
think about adding this to TokenFilter, that passes to the delegated Tokenizer (using instanceof

> Can't specify AttributeSource for Tokenizer
> -------------------------------------------
>                 Key: LUCENE-1804
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Bug
>            Reporter: Yonik Seeley
>         Attachments: LUCENE-1804.patch
> One can't currently specify the attribute source for a Tokenizer like one can with any
other TokenStream.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message