Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of rik@attivio.com designates
 207.46.163.183 as permitted sender)
From: Rik Tamm-Daniels <rik@attivio.com>
To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>
Subject: Re: ICUTokenizer acting very strangely with oriental characters
Thread-Topic: ICUTokenizer acting very strangely with oriental characters
Thread-Index: AQHPtndVU/g6E9mGYUOO2fArqaMe4pvNlV0AgAAP8xY=
Date: Tue, 12 Aug 2014 23:57:07 +0000
Message-ID: 
 <e84a4e9aafaa4ebabc5d32b0c65422d3@CO1PR05MB409.namprd05.prod.outlook.com>
References: <53EA8BD1.3060202@elyograg.org>,<53EA9C72.80002@elyograg.org>
In-Reply-To: <53EA9C72.80002@elyograg.org>
Accept-Language: en-US
Content-Language: en-US
Content-Type: multipart/alternative;
	boundary="_000_e84a4e9aafaa4ebabc5d32b0c65422d3CO1PR05MB409namprd05pro_"
MIME-Version: 1.0

--_000_e84a4e9aafaa4ebabc5d32b0c65422d3CO1PR05MB409namprd05pro_
Content-Type: text/plain; charset="iso-2022-jp"
Content-Transfer-Encoding: quoted-printable

mmn

jnbbbjb)nkkkk9nooooooon

Sent from my HTC

----- Reply message -----
From: "Shawn Heisey" <solr@elyograg.org>
To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>
Subject: ICUTokenizer acting very strangely with oriental characters
Date: Tue, Aug 12, 2014 19:00

See the original message on this thread for full details.  Some
additional information:

This happens on version 4.6.1, 4.7.2, and 4.9.0.  Here is a screenshot
showing the analysis problem in more detail.  The first line you can see
is the ICUTokenizer.

https://www.dropbox.com/s/9wbi7lz77ivya9j/ICUTokenizer-wrong-analysis.png

The original field value was:

=1B$B#2#0@$5*$N#1#0#0?M=1B(B;=1B$B%]!<%H%l!<%H%"!<%+%$%V%9=1B(B;=1B$B@/<#2H=
!&73?M=1B(B;=1B$B@/<#2H!&;XF3=1B(B
=1B$B<T!&73?M=1B(B;[=1B$B@/=1B(B =1B$B<#=1B(B],100peopeof20century,pploftwe=
ntycentury,pploftwentycentury

Thanks,
Shawn


--_000_e84a4e9aafaa4ebabc5d32b0c65422d3CO1PR05MB409namprd05pro_--