Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of solr@elyograg.org designates
 166.70.79.219 as permitted sender)
Message-ID: <53EBA62F.8060507@elyograg.org>
Date: Wed, 13 Aug 2014 11:53:51 -0600
From: Shawn Heisey <solr@elyograg.org>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64;
 rv:31.0) Gecko/20100101 Thunderbird/31.0
MIME-Version: 1.0
To: solr-user@lucene.apache.org
Subject: Re: ICUTokenizer acting very strangely with oriental characters
References: <53EA8BD1.3060202@elyograg.org>	<53EA9C72.80002@elyograg.org>
	<1892F28C-D8CA-4769-A713-29146BE8563D@gmail.com>
	<53EAB8FB.5010606@elyograg.org>
 <CALvfXiJ6NvBKO2KuhvAAb71eVACuW_snPU-oiTivWthDZ2hJKg@mail.gmail.com>
In-Reply-To: 
 <CALvfXiJ6NvBKO2KuhvAAb71eVACuW_snPU-oiTivWthDZ2hJKg@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

On 8/12/2014 9:13 PM, Steve Rowe wrote:
> In the table below, the "IsSameS" (is same script) and "SBreak?" (script
> break = not IsSameS) decisions are based on what I mentioned in my previous
> message, and the "WBreak" (word break) decision is based on UAX#29 word
> break rules:
>
> Char    Code Point   Script        IsSameS?    SBreak?  WBreak?
> ------    --------------   -------        -------------    ---------
> -----------
> 治        U+6CBB       Han          Yes              No            Yes
> ]          U+005D        Common   Yes              No            Yes
> ,          U+002C        Common   Yes              No            Yes
> 1         U+0031         Common   --                 --              --
>
> First, script boundaries are found and used as token boundaries - in the
> above case, no script boundary is found between "治" and "1" - and then
> UAX#29 word break rules are used to find token boundaries inbetween script
> boundaries - in the above case, there are word boundaries between each
> character, but ICUTokenizer throws away punctuation-only sequences between
> token boundaries.

What should we use as a dividing character for situations like this? 
Should we tell our customer that they can't start keywords like this
(for searching/filtering) with a number?

Thanks,
Shawn