lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Smiley (JIRA)" <>
Subject [jira] [Created] (LUCENE-3366) StandardFilter only works with ClassicTokenizer and only when version < 3.1
Date Mon, 08 Aug 2011 22:18:27 GMT
StandardFilter only works with ClassicTokenizer and only when version < 3.1

                 Key: LUCENE-3366
             Project: Lucene - Java
          Issue Type: Improvement
          Components: modules/analysis
    Affects Versions: 3.3
            Reporter: David Smiley

The StandardFilter used to remove periods from acronyms and apostrophes-S's where they occurred.
And it used to work in conjunction with the StandardTokenizer.  Presently, it only does this
with ClassicTokenizer and when the lucene match version is before 3.1. Here is a excerpt from
the code:
  public final boolean incrementToken() throws IOException {
    if (matchVersion.onOrAfter(Version.LUCENE_31))
      return input.incrementToken(); // TODO: add some niceties for the new grammar
      return incrementTokenClassic();

It seems to me that in the great refactor of the standard tokenizer, LUCENE-2167, something
was forgotten here. I think that if someone uses the ClassicTokenizer then no matter what
the version is, this filter should do what it used to do. And the TODO suggests someone forgot
to make this filter do something useful for the StandardTokenizer.  Or perhaps that idea should
be discarded and this class should be named ClassicTokenFilter.

In any event, the javadocs for this class appear out of date as there is no mention of ClassicTokenizer,
and the wiki is out of date too.

This message is automatically generated by JIRA.
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message