lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Male (JIRA)" <>
Subject [jira] [Updated] (LUCENE-3396) Make TokenStream Reuse Mandatory for Analyzers
Date Fri, 26 Aug 2011 05:43:29 GMT


Chris Male updated LUCENE-3396:

    Attachment: LUCENE-3396-rab.patch

This is the uber patch which converts all but 8 Analyzers over to using ReusableAnalyzerBase.

This also introduces the ReuseStrategy concept to ReusableAnalyzerBase (will need some documentation
before committing).  For simplicity, the ReuseStrategy maintains its own CloseableThreadLocal.

The remaining Analyzers can be converted when I push the ReuseAnalyzerBase logic into Analyzer.

After some small tidy (and a review) I think we should commit this even though we're going
to collapse ReusabeAnalyzerBase.

> Make TokenStream Reuse Mandatory for Analyzers
> ----------------------------------------------
>                 Key: LUCENE-3396
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: modules/analysis
>            Reporter: Chris Male
>         Attachments: LUCENE-3396-rab.patch
> In LUCENE-2309 it became clear that we'd benefit a lot from Analyzer having to return
reusable TokenStreams.  This is a big chunk of work, but its time to bite the bullet.
> I plan to attack this in the following way:
> - Collapse the logic of ReusableAnalyzerBase into Analyzer
> - Add a ReuseStrategy abstraction to Analyzer which controls whether the TokenStreamComponents
are reused globally (as they are today) or per-field.
> - Convert all Analyzers over to using TokenStreamComponents.  I've already seen that
some of the TokenStreams created in tests need some work to be reusable (even if they aren't
> - Remove Analyzer.reusableTokenStream and convert everything over to using .tokenStream
(which will now be returning reusable TokenStreams).

This message is automatically generated by JIRA.
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message