lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler (JIRA)" <>
Subject [jira] Commented: (LUCENE-1826) All Tokenizer implementations should have constructors that take AttributeSource and AttributeFactory
Date Fri, 21 Aug 2009 05:34:14 GMT


Uwe Schindler commented on LUCENE-1826:

bq. without the Tokenizer.reset(Reader, AttributeSource), i won't be able to reuse Tokenizer
instances (will have to create a fresh one each time)

This is not possible per design. The AttributeSource cannot be changed. It is created during
creation of the classes (this is why it is in the ctor and nowhere else). For filters, the
attributes come from the input token stream.

bq. Is the reflection penalty on the new TokenStream stuff incurred per root AttributeSource?,
or per TokenFilter/TokenStream?

The reflection penalty is one-time per class (because of static cache of "known" classes),
so all attributeimpl are inspected one time when a new AttributeSouce like TokenStream is
created. There is an additional reflection cost, when new attributes are added, but also one
time per AttributeImpl class.
Since the last changes in TokenStream the reflection is therefore no longer a penalty. The
only problem is more work to construct an TokenStream (filling the LinkedHashMaps), because
of that you should reuse TokenStream-chains.

bq. that is, if i pass the same AttributeSource to 10 TokenStreams, is the reflection cost
the same as if i passed it to just one?

No change!

> All Tokenizer implementations should have constructors that take AttributeSource and
> -----------------------------------------------------------------------------------------------------
>                 Key: LUCENE-1826
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Analysis
>    Affects Versions: 2.9
>            Reporter: Tim Smith
>             Fix For: 2.9
> I have a TokenStream implementation that joins together multiple sub TokenStreams (i
then do additional filtering on top of this, so i can't just have the indexer do the merging)
> in 2.4, this worked fine.
> once one sub stream was exhausted, i just started using the next stream 
> however, in 2.9, this is very difficult, and requires copying Term buffers for every
token being aggregated
> however, if all the sub TokenStreams share the same AttributeSource, and my "concat"
TokenStream shares the same AttributeSource, this goes back to being very simple (and very
> So for example, i would like to see the following constructor added to StandardTokenizer:
> {code}
>   public StandardTokenizer(AttributeSource source, Reader input, boolean replaceInvalidAcronym)
>     super(source);
>     ...
>   }
> {code}
> would likewise want similar constructors added to all Tokenizer sub classes provided
by lucene

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message