lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Cheng (JIRA)" <j...@apache.org>
Subject [jira] Created: (LUCENE-1490) CJKTokenizer convert HALFWIDTH_AND_FULLWIDTH_FORMS wrong
Date Fri, 12 Dec 2008 06:04:46 GMT
CJKTokenizer convert   HALFWIDTH_AND_FULLWIDTH_FORMS wrong
----------------------------------------------------------

                 Key: LUCENE-1490
                 URL: https://issues.apache.org/jira/browse/LUCENE-1490
             Project: Lucene - Java
          Issue Type: Bug
            Reporter: Daniel Cheng
             Fix For: 2.4


CJKTokenizer have these lines..
                if (ub == Character.UnicodeBlock.HALFWIDTH_AND_FULLWIDTH_FORMS) {
                    /** convert  HALFWIDTH_AND_FULLWIDTH_FORMS to BASIC_LATIN */
                    int i = (int) c;
                    i = i - 65248;
                    c = (char) i;
                }

This is wrong. Some character in the block (e.g. U+ff68) have no BASIC_LATIN counterparts.
Only 65281-65374 can be converted this way.

The fix is

             if (ub == Character.UnicodeBlock.HALFWIDTH_AND_FULLWIDTH_FORMS && i <=
65474 && i> 65281) {
                    /** convert  HALFWIDTH_AND_FULLWIDTH_FORMS to BASIC_LATIN */
                    int i = (int) c;
                    i = i - 65248;
                    c = (char) i;
                }

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message