lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Kaelbling <>
Subject RE: CachingTokenFilter extensibility and LUCENE-1685
Date Fri, 28 Aug 2009 15:54:17 GMT
Hi Uwe,

The problem is that I need to have a random access token stream for other reasons, and don't
want CachingTokenFilter to buffer up a redundant copy of it.  In existing releases I subclass
it to override all the methods to use my store, and ignore the LinkedList cache member.  The
old internal structures were still present, but were never used.  In 2.9 I can't do that any
more, and without a subclassed object I have no way to prevent WeightedSpanTermExtractor from
wrapping the stream.

If there were some way to tell WeightedSpanTermExtractor not wrap the stream (a new TokenStream.isCachingTokens()
method, checking for an new "CachedTokenStream" interface rather than for CachingTokenFilter,
some attribute, anything! :-) then I could still work with the public API.

  - David

David Kaelbling
Senior Software Engineer
Black Duck Software, Inc.
T +1.781.810.2041
F +1.781.891.5145
From: Uwe Schindler []
Sent: Friday, August 28, 2009 4:03 AM
Subject: RE: CachingTokenFilter extensibility and LUCENE-1685

Hi David,

What is exactly your problem? Even the old 2.4 CachingTokenFilter did not
expose its internal structures, so overriding would not change its internal
implementation. The only change now is, that *all* TokenFilters in core have
final implementations, which is a consequence of the new TokenStream API and
the migration path to it. So it should not be possible to override
next()/next(Token)/incrementToken() in all TokenStreams, as extensibility of
the whole API is because of simply adding new TokenFilters into the chain,
that do what you want to add. Let users override incrementToken() would
possibly break a lot of things (see LUCENE-1753)

To fix your specific problems, it may be an idea to add a method
(isCachingTokens) in future to TokenStreams that default to false and is
true for CachingTokenFilter and TeeSinkTokenStream.SinkTokenStream.
Highlighter would be able to detect, if it can reset() (better name would be
rewind) the TokenStream. In this case you could simply provide another
TokenFilter subclass with isCachingTokens=true and random access to the

Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen

> -----Original Message-----
> From: David Kaelbling []
> Sent: Thursday, August 27, 2009 10:40 PM
> To:
> Subject: CachingTokenFilter extensibility and LUCENE-1685
> Hi,
> Looking at Lucene 2.9 trunk, CachingTokenFilter seems much less extensible
> than before.  In previous releases I subclassed it so I could back the
> cache with an array and provide random access to the stream.  I can't see
> how to do this any more, and the
> WeightedSpanTermExtractor.getReaderForField() is still hardwired to
> require a CachingTokenFilter-derived object.
> Am I missing something?  Having two copies of the token stream, one for
> random access and one hidden inside the CachingTokenFilter, does not sound
> efficient :-)
>   Thanks,
>   David
> --
> David Kaelbling
> Senior Software Engineer
> Black Duck Software, Inc.
> T +1.781.810.2041
> F +1.781.891.5145
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message