lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steven Rowe (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-1545) Standard analyzer does not correctly tokenize combining character U+0364 COMBINING LATIN SMALL LETTRE E
Date Wed, 29 Sep 2010 05:52:35 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12916033#action_12916033
] 

Steven Rowe commented on LUCENE-1545:
-------------------------------------

I updated AnalyzerTest.java:

{code:java}
import java.io.FileOutputStream;
import java.io.OutputStreamWriter;
import java.io.StringReader;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.util.Version;

public class AnalyzerTest {
  public static void test() throws Exception {
    StandardAnalyzer a = new StandardAnalyzer(Version.LUCENE_31);
    TokenStream ts = a.tokenStream("", new StringReader("moͤchte m mo\u0364chte "));
    OutputStreamWriter writer = new OutputStreamWriter(new FileOutputStream("output.txt"),
"UTF-8");
    while (ts.incrementToken()) {
      writer.append(ts.toString()).append(System.getProperty("line.separator"));
    }
    writer.flush();
    writer.close();
  }

  public static void main(String[] argv) throws Exception {
    test();
  }
}
{code}

Here's what goes into output.txt when I compile {{AnalyzerTest.java}} with {{javac -encoding
UTF-8 -cp lucene/dev/branches/branch_3x/lucene/build/lucene-core-3.1-SNAPSHOT.jar" AnalyzerTest}}:
{code:java}
(moͤchte,startOffset=0,endOffset=7,positionIncrement=1,type=<ALPHANUM>)
(m,startOffset=8,endOffset=9,positionIncrement=1,type=<ALPHANUM>)
(moͤchte,startOffset=10,endOffset=17,positionIncrement=1,type=<ALPHANUM>)
{code}

With LUCENE-2167 committed on the 3.X branch and on trunk, I think this issue is resolved.
 Please reopen if you see different behavior.

> Standard analyzer does not correctly tokenize combining character U+0364 COMBINING LATIN
SMALL LETTRE E
> -------------------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-1545
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1545
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: contrib/analyzers
>    Affects Versions: 2.4
>         Environment: Linux x86_64, Sun Java 1.6
>            Reporter: Andreas Hauser
>            Priority: Minor
>             Fix For: 3.1, 4.0
>
>         Attachments: AnalyzerTest.java
>
>
> Standard analyzer does not correctly tokenize combining character U+0364 COMBINING LATIN
SMALL LETTRE E.
> The word "moͤchte" is incorrectly tokenized into "mo" "chte", the combining character
is lost.
> Expected result is only on token "moͤchte".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message