lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Muir (JIRA)" <>
Subject [jira] Commented: (LUCENE-2034) Massive Code Duplication in Contrib Analyzers - unifly the analyzer ctors
Date Sun, 08 Nov 2009 20:15:32 GMT


Robert Muir commented on LUCENE-2034:

Simon, i started looking at this, the testStemExclusionTable( for BrazilianAnalyzer is actually
not related to stopwords and should not be changed.

BrazilianAnalyzer has a .setStemExclusionTable() method that allows you to supply a set of
words that should not be stemmed. 

This test is to ensure  that if you change the stem exclusion table with this method, that
reusableTokenStream will force the creation of a new BrazilianStemFilter with this modified
exclusion table so that it will take effect immediately, the way it did with .tokenStream()
before this analyzer supported reusableTokenStream()

> Massive Code Duplication in Contrib Analyzers - unifly the analyzer ctors
> -------------------------------------------------------------------------
>                 Key: LUCENE-2034
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: contrib/analyzers
>    Affects Versions: 2.9
>            Reporter: Simon Willnauer
>            Priority: Minor
>             Fix For: 3.0
>         Attachments: LUCENE-2034.patch, LUCENE-2034.patch, LUCENE-2034.patch, LUCENE-2034.patch
> Due to the variouse tokenStream APIs we had in lucene analyzer subclasses need to implement
at least one of the methodes returning a tokenStream. When you look at the code it appears
to be almost identical if both are implemented in the same analyzer.  Each analyzer defnes
the same inner class (SavedStreams) which is unnecessary.
> In contrib almost every analyzer uses stopwords and each of them creates his own way
of loading them or defines a large number of ctors to load stopwords from a file, set, arrays
etc.. those ctors should be removed / deprecated and eventually removed.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message