lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <>
Subject RE: who clears attributes?
Date Mon, 10 Aug 2009 17:50:48 GMT
Yes. Is there a way to enforce this for all Tokenizers automatically? As
incrementToken() will be abstract in 3.0, there cannot be a default impl. So
all Tokenizers should call clearAttributes() as first call in

Then we have still the problem of the slow iterator creation (which was
speed up a little bit by removing the unmodifiable wrapper). This can be
solved by using an additional ArrayList in AttributeSource that gets all
AttributeImpl instances, but this would bring an additional initialization
cost() on creating the Tokenizer chain.

Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen

> -----Original Message-----
> From: [] On Behalf Of Yonik
> Seeley
> Sent: Monday, August 10, 2009 7:42 PM
> To:
> Subject: Re: who clears attributes?
> Thinking through this a little more, I don't see an alternative to the
> tokenizer clearing all attributes at the start of incrementToken().
> Consider a DefaultPayloadTokenFilter that only sets a payload if one
> isn't already set - it's clear that this filter can't clear the
> payload attribute, so it must be cleared by the head of the chain -
> the tokenizer.  Right?
> -Yonik
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message