lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grant Ingersoll <>
Subject Re: attribute thoughts
Date Thu, 13 Aug 2009 15:21:13 GMT

On Aug 13, 2009, at 10:29 AM, Yonik Seeley wrote:

> I'm liking the new attribute based analysis (in conjunction with
> reusability), but I'm running into some questions...
> Is it valid for tokenizers or token filters add new attributes after
> their constructor (after they have processed some tokens)?

I don't know if it explicitly prohibits it, but I believe the docs  
discourage it for performance reasons.

> Should restoreState() be able to add attributes (it currently throws
> an exception)?  If not, does that mean that it's not supported/advised
> to use state across different TokenStreams?

Not sure

> We've previously seen that the native java clone() can be much slower
> than implementing it ourselves in Java.  Should we have our own
> clone() method on Attribute?  Or just implement clone() ourselves and
> require that subclasses override if needed?  This is inner-loop
> per-token stuff, and a single captureState() will invoke many clone
> operations (6 attributes make up the legacy Token object).

Sounds reasonable, but I haven't benchmarked.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message