Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Date: Wed, 1 Feb 2012 12:48:58 +0000 (UTC)
From: "Robert Muir (Commented) (JIRA)" <jira@apache.org>
To: dev@lucene.apache.org
Message-ID: 
 <135911455.2459.1328100538678.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <1967735086.64813.1327256679825.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (SOLR-3056) Introduce Japanese field type in
 schema.xml
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/SOLR-3056?page=3Dcom.atlassian.=
jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D13197=
802#comment-13197802 ]=20

Robert Muir commented on SOLR-3056:
-----------------------------------

Ill do the svn moves portion of this, so that we can iterate on the default=
 configuration etc.

I think what Christian defined in the description is the way to go as a sta=
rt...
               =20
> Introduce Japanese field type in schema.xml
> -------------------------------------------
>
>                 Key: SOLR-3056
>                 URL: https://issues.apache.org/jira/browse/SOLR-3056
>             Project: Solr
>          Issue Type: New Feature
>          Components: Schema and Analysis
>    Affects Versions: 3.6, 4.0
>            Reporter: Christian Moen
>
> Kuromoji (LUCENE-3305) is now on both on trunk and branch_3x (thanks agai=
n Robert, Uwe and Simon). It would be very good to get a default field type=
 defined for Japanese in {{schema.xml}} so we can good Japanese out-of-the-=
box support in Solr.
> I've been playing with the below configuration today, which I think is a =
reasonable starting point for Japanese.  There's lot to be said about vario=
us considerations necessary when searching Japanese, but perhaps a wiki pag=
e is more suitable to cover the wider topic?
> In order to make the below {{text_ja}} field type work, Kuromoji itself a=
nd its analyzers need to be seen by the Solr classloader.  However, these a=
re currently in contrib and I'm wondering if we should consider moving them=
 to core to make them directly available.  If there are concerns with addit=
ional memory usage, etc. for non-Japanese users, we can make sure resources=
 are loaded lazily and only when needed in factory-land.
> Any thoughts?
> {code:xml}
> <!-- Text field type is suitable for Japanese text using morphological an=
alysis
>      NOTE: Please copy files
>        contrib/analysis-extras/lucene-libs/lucene-kuromoji-x.y.z.jar
>        dist/apache-solr-analysis-extras-x.y.z.jar
>      to your Solr lib directory (i.e. example/solr/lib) before before sta=
rting Solr.
>      (x.y.z refers to a version number)
>      If you would like to optimize for precision, default operator AND wi=
th
>        <solrQueryParser defaultOperator=3D"AND"/>
>      below (this file).  Use "OR" if you would like to optimize for recal=
l (default).
> -->
> <fieldType name=3D"text_ja" class=3D"solr.TextField" positionIncrementGap=
=3D"100" autoGeneratePhraseQueries=3D"false">
>   <analyzer>
>     <!-- Kuromoji Japanese morphological analyzer/tokenizer
>          Use search-mode to get a noun-decompounding effect useful for se=
arch.
>          Example:
>            =E9=96=A2=E8=A5=BF=E5=9B=BD=E9=9A=9B=E7=A9=BA=E6=B8=AF (Kansai=
 International Airpart) becomes =E9=96=A2=E8=A5=BF (Kansai) =E5=9B=BD=E9=9A=
=9B (International) =E7=A9=BA=E6=B8=AF (airport)
>            so we get a match for =E7=A9=BA=E6=B8=AF (airport) as we would=
 expect from a good search engine
>          Valid values for mode are:
>             normal: default segmentation
>             search: segmentation useful for search (extra compound splitt=
ing)
>           extended: search mode with unigramming of unknown words (experi=
mental)
>          NOTE: Search mode improves segmentation for search at the expens=
e of part-of-speech accuracy
>     -->
>     <tokenizer class=3D"solr.KuromojiTokenizerFactory" mode=3D"search"/>
>     <!-- Reduces inflected verbs and adjectives to their base/dectionary =
forms (=E8=BE=9E=E6=9B=B8=E5=BD=A2) -->=09
>     <filter class=3D"solr.KuromojiBaseFormFilterFactory"/>
>     <!-- Optionally remove tokens with certain part-of-speeches
>     <filter class=3D"solr.KuromojiPartOfSpeechStopFilterFactory" tags=3D"=
stopTags.txt" enablePositionIncrements=3D"true"/> -->
>     <!-- Normalizes full-width romaji to half-with and half-width kana to=
 full-width (Unicode NFKC subset) -->
>     <filter class=3D"solr.CJKWidthFilterFactory"/>
>     <!-- Lower-case romaji characters -->
>     <filter class=3D"solr.LowerCaseFilterFactory"/>
>   </analyzer>
> </fieldType>
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrato=
rs: https://issues.apache.org/jira/secure/ContactAdministrators!default.jsp=
a
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org