Return-Path: 
 <solr-dev-return-16441-apmail-lucene-solr-dev-archive=lucene.apache.org@lucene.apache.org>
Delivered-To: apmail-lucene-solr-dev-archive@minotaur.apache.org
Received: (qmail 3260 invoked from network); 5 May 2009 21:39:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 May 2009 21:39:58 -0000
Received: (qmail 45298 invoked by uid 500); 5 May 2009 21:40:07 -0000
Delivered-To: apmail-lucene-solr-dev-archive@lucene.apache.org
Received: (qmail 45270 invoked by uid 500); 5 May 2009 21:40:07 -0000
Mailing-List: contact solr-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:solr-dev-help@lucene.apache.org>
List-Unsubscribe: <mailto:solr-dev-unsubscribe@lucene.apache.org>
List-Post: <mailto:solr-dev@lucene.apache.org>
List-Id: <solr-dev.lucene.apache.org>
Reply-To: solr-dev@lucene.apache.org
Delivered-To: mailing list solr-dev@lucene.apache.org
Received: (qmail 45237 invoked by uid 99); 5 May 2009 21:40:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 May 2009 21:40:06 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 May 2009 21:39:57 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 69016234C051
	for <solr-dev@lucene.apache.org>; Tue,  5 May 2009 14:39:36 -0700 (PDT)
Message-ID: <25959493.1241559576429.JavaMail.jira@brutus>
Date: Tue, 5 May 2009 14:39:36 -0700 (PDT)
From: "Yonik Seeley (JIRA)" <jira@apache.org>
To: solr-dev@lucene.apache.org
Subject: [jira] Commented: (SOLR-1078) WordDelimiterFilter do wrong word
 breaking for Thai vowel
In-Reply-To: <207273798.1237553330726.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/SOLR-1078?page=3Dcom.atlassian.=
jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D12706=
211#action_12706211 ]=20

Yonik Seeley commented on SOLR-1078:
------------------------------------

Are these characters all in the basic multilingual plane?

Here is the relevant code how WordDelimiterFilter characterizes chars:

{code}
  [...]
    } else if (Character.isLowerCase(ch)) {
      return LOWER;
    } else if (Character.isLetter(ch)) {
      return UPPER;
    } else {
      return SUBWORD_DELIM;
    }
{code}



> WordDelimiterFilter do wrong word breaking for Thai vowel
> ---------------------------------------------------------
>
>                 Key: SOLR-1078
>                 URL: https://issues.apache.org/jira/browse/SOLR-1078
>             Project: Solr
>          Issue Type: Bug
>          Components: Analysis
>    Affects Versions: 1.4
>         Environment: Ubuntu 8.10 64bit
> Java 1.6.0_10
>            Reporter: SIriwat Aumngamsup
>
> With any configuration of schema.xml
> {code:xml}<filter class=3D"solr.WordDelimiterFilterFactory" />{code}
> will do wrong word breaking with Thai characters.
> ----
> Example: "=E0=B8=9C=E0=B8=B9=E0=B9=89 =E0=B9=83=E0=B8=AB=E0=B8=8D=E0=B9=
=88 =E0=B8=9A=E0=B9=89=E0=B8=B2=E0=B8=99"
> Wrong result: 0 =3D> "=E0=B8=9C", 1 =3D> "=E0=B9=83=E0=B8=AB=E0=B8=8D", 2=
 =3D> "=E0=B8=9A", 3 =3D> "=E0=B8=B2=E0=B8=99"
> Expect result: 0 =3D> "=E0=B8=9C=E0=B8=B9=E0=B9=89", 1 =3D> "=E0=B9=83=E0=
=B8=AB=E0=B8=8D=E0=B9=88", 2 =3D> "=E0=B8=9A=E0=B9=89=E0=B8=B2=E0=B8=99"
> ----
> Example2: "=E0=B8=9C=E0=B8=B9=E0=B9=89=E0=B9=83=E0=B8=AB=E0=B8=8D=E0=B9=
=88=E0=B8=9A=E0=B9=89=E0=B8=B2=E0=B8=99" (no space)
> Wrong result: 0 =3D> "=E0=B8=9C", 1 =3D> "=E0=B9=83=E0=B8=AB=E0=B8=8D", 2=
 =3D> "=E0=B8=9A", 3 =3D> "=E0=B8=B2=E0=B8=99" (same result)
> Expect result: 0 =3D> "=E0=B8=9C=E0=B8=B9=E0=B9=89=E0=B9=83=E0=B8=AB=E0=
=B8=8D=E0=B9=88=E0=B8=9A=E0=B9=89=E0=B8=B2=E0=B8=99"
> ----
> There's a similar problem with Drupal (http://drupal.org/node/335928)

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.