Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Date: Tue, 9 Aug 2011 01:58:27 +0000 (UTC)
From: "David Smiley (JIRA)" <jira@apache.org>
To: dev@lucene.apache.org
Message-ID: 
 <711008040.18655.1312855107110.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <1643962731.18073.1312841907212.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (LUCENE-3366) StandardFilter only works with
 ClassicTokenizer and only when version < 3.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/LUCENE-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081363#comment-13081363 ] 

David Smiley commented on LUCENE-3366:
--------------------------------------

Doh! Yes, I didn't notice it, Rob.  But still... the purpose of StandardFilter in its current state seems to only exist to satisfy backwards compatibility for code that uses Lucene at a pre 3.x era; nothing more.  Shouldn't it be marked @Deprecated to warn people?.  Or, the "TODO" should be done to do something. However the current StandardTokenizer doesn't really have equivalent token types to ClassicTokenizer in order for StandardFilter to actually do something useful. So then there is no TODO to do.

> StandardFilter only works with ClassicTokenizer and only when version < 3.1
> ---------------------------------------------------------------------------
>
>                 Key: LUCENE-3366
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3366
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: modules/analysis
>    Affects Versions: 3.3
>            Reporter: David Smiley
>
> The StandardFilter used to remove periods from acronyms and apostrophes-S's where they occurred. And it used to work in conjunction with the StandardTokenizer.  Presently, it only does this with ClassicTokenizer and when the lucene match version is before 3.1. Here is a excerpt from the code:
> {code:lang=java}
>   public final boolean incrementToken() throws IOException {
>     if (matchVersion.onOrAfter(Version.LUCENE_31))
>       return input.incrementToken(); // TODO: add some niceties for the new grammar
>     else
>       return incrementTokenClassic();
>   }
> {code}
> It seems to me that in the great refactor of the standard tokenizer, LUCENE-2167, something was forgotten here. I think that if someone uses the ClassicTokenizer then no matter what the version is, this filter should do what it used to do. And the TODO suggests someone forgot to make this filter do something useful for the StandardTokenizer.  Or perhaps that idea should be discarded and this class should be named ClassicTokenFilter.
> In any event, the javadocs for this class appear out of date as there is no mention of ClassicTokenizer, and the wiki is out of date too.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org