lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yonik Seeley <>
Subject attribute thoughts
Date Thu, 13 Aug 2009 14:29:46 GMT
I'm liking the new attribute based analysis (in conjunction with
reusability), but I'm running into some questions...

Is it valid for tokenizers or token filters add new attributes after
their constructor (after they have processed some tokens)?

Should restoreState() be able to add attributes (it currently throws
an exception)?  If not, does that mean that it's not supported/advised
to use state across different TokenStreams?

We've previously seen that the native java clone() can be much slower
than implementing it ourselves in Java.  Should we have our own
clone() method on Attribute?  Or just implement clone() ourselves and
require that subclasses override if needed?  This is inner-loop
per-token stuff, and a single captureState() will invoke many clone
operations (6 attributes make up the legacy Token object).


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message