lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yonik Seeley <>
Subject Re: attribute thoughts
Date Fri, 14 Aug 2009 16:23:54 GMT
On Thu, Aug 13, 2009 at 4:32 PM, Michael Busch<> wrote:
> On 8/13/09 7:29 AM, Yonik Seeley wrote:
>> I'm liking the new attribute based analysis (in conjunction with
>> reusability), but I'm running into some questions...
>> Is it valid for tokenizers or token filters add new attributes after
>> their constructor (after they have processed some tokens)?
> At the moment we're saying in the javadocs of TokenStream that all
> Attributes should be
> added up front.

Hmmm, OK... in which case, token producers using restoreState() would
not have to call clearAttributes() first.

> We could change these semantics. I had some thoughts about
> it in the original
> JIRA issue (LUCENE-1422).

Apologies if I'm rehashing anything - it's hard to keep up with some
of those monster (high volume) issues.

> So back to your question if we should allow restoreState() to add attributes
> and use a state across different AttributeSources: the complication is that we can only
> allow that if  the different AttributeSource were filled using the same AttributeFactory,
> different AtttributeImpls could be in the sources and the copying wouldn't
> work anymore.

Hmmm, so perhaps just an assertion that the factories are equal... and
documentation saying that moving state from one stream to the other
requires identical factories?  Anyway, I don't currently have a use
case for this... I was just wondering.

Another thing I was wondering about was the opacity of State - one
can't inspect or change the attributes w/o restoring it first.
Undesirable limitation, or feature allowing more flexible state


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message