Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Message-ID: <16342584.90111289262606925.JavaMail.jira@thor>
Date: Mon, 8 Nov 2010 19:30:06 -0500 (EST)
From: "Robert Muir (JIRA)" <jira@apache.org>
To: dev@lucene.apache.org
Subject: [jira] Commented: (SOLR-2211) Create Solr FilterFactory for Lucene
 StandardTokenizer with  UAX#29 support
In-Reply-To: <6255994.176671288632744834.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/SOLR-2211?page=3Dcom.atlassian.=
jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D12929=
849#action_12929849 ]=20

Robert Muir commented on SOLR-2211:
-----------------------------------

Great, I look forward to the results.

By the way, on SOLR-2210 i also added the ICU filters, you could consider r=
eplacing LowerCaseFilterFactory with ICUNormalizer2Factory (just use the de=
faults).
In addition to better lowercasing (e.g. =C3=9F -> ss), this would also brin=
g the advantages described in http://unicode.org/reports/tr15/

Alternatively, if you are already using both LowerCaseFilterFactory and ASC=
IIFoldingFilterFactory, you can replace both with ICUFoldingFilterFactory,
which goes further and also incorporates http://www.unicode.org/reports/tr3=
0/tr30-4.html


> Create Solr FilterFactory for Lucene StandardTokenizer with  UAX#29 suppo=
rt
> -------------------------------------------------------------------------=
--
>
>                 Key: SOLR-2211
>                 URL: https://issues.apache.org/jira/browse/SOLR-2211
>             Project: Solr
>          Issue Type: New Feature
>    Affects Versions: 3.1
>            Reporter: Tom Burton-West
>            Assignee: Robert Muir
>            Priority: Minor
>             Fix For: 3.1, 4.0
>
>         Attachments: SOLR-2211.patch
>
>
> The Lucene 3.x StandardTokenizer with UAX#29 support provides benefits fo=
r non-English tokenizing.  Presently it can be invoked by using the Standar=
dTokenizerFactory and setting the Version to 3.1.  However, it would be use=
ful to be able to use the improved unicode processing without necessarily i=
ncluding the ip address and email address processing of StandardAnalyzer.  =
 A FilterFactory that allowed the use of the StandardTokenizer with UAX#29 =
support on its own would be useful.

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org