lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Filip Svendsen (JIRA)" <j...@apache.org>
Subject [jira] [Created] (LUCENE-3038) DictionaryCompoundWordTokenFilter fails to create some tokens for final parts of words
Date Tue, 19 Apr 2011 21:48:05 GMT
DictionaryCompoundWordTokenFilter fails to create some tokens for final parts of words
--------------------------------------------------------------------------------------

                 Key: LUCENE-3038
                 URL: https://issues.apache.org/jira/browse/LUCENE-3038
             Project: Lucene - Java
          Issue Type: Bug
          Components: Analysis
    Affects Versions: 3.1, 4.0
            Reporter: Filip Svendsen
             Fix For: 4.0, 3.1


DictionaryCompoundWordTokenFilter: Due to an off-by-one error, a word component placed last
in a compound word, will not get a token if its length is equal to the minimal sub-word length.

Example:
min sub-word length: 4
Dictionary: {"alfa", "beta"}
word: "alfabeta"
Created tokens: {"alfabeta", "alfa"}
Expected tokens: {"alfabeta", "alfa", "beta"}

I have a patch with a testcase that fails on versions 3.1 and 4.0 (probably for everything
between as well, and for previous versions), along with a bugfix.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message