lucene-solr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Koji Sekiguchi (JIRA)" <j...@apache.org>
Subject [jira] Updated: (SOLR-822) CharFilter - normalize characters before tokenizer
Date Thu, 23 Oct 2008 15:04:44 GMT

     [ https://issues.apache.org/jira/browse/SOLR-822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Koji Sekiguchi updated SOLR-822:
--------------------------------

    Attachment: sample_mapping_ja.txt

sample_mapping_ja.txt file is attached. To use it, open schema.xml by editor and define textCharNorm
fieldType and text_cjk field as follows:

{code:xml}
<fieldType name="textCharNorm" class="solr.TextField" positionIncrementGap="100" >
  <analyzer>
    <charFilter class="solr.MappingCharFilterFactory" mapping="sample_mapping_ja.txt"/>
    <tokenizer class="solr.MappingCJKTokenizerFactory"/>
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
</fieldType>
{code}

{code:xml}
<field name="text_cjk" type="textCharNorm" indexed="true" stored="true"/>
{code}

then start solr and access analysis.jsp.

> CharFilter - normalize characters before tokenizer
> --------------------------------------------------
>
>                 Key: SOLR-822
>                 URL: https://issues.apache.org/jira/browse/SOLR-822
>             Project: Solr
>          Issue Type: New Feature
>          Components: Analysis
>            Reporter: Koji Sekiguchi
>            Priority: Minor
>         Attachments: character-normalization.JPG, sample_mapping_ja.txt, SOLR-822.patch,
SOLR-822.patch
>
>
> A new plugin which can be placed in front of <tokenizer/>.
> {code:xml}
> <fieldType name="textCharNorm" class="solr.TextField" positionIncrementGap="100" >
>   <analyzer>
>     <charFilter class="solr.MappingCharFilterFactory" mapping="mapping_ja.txt" />
>     <tokenizer class="solr.MappingCJKTokenizerFactory"/>
>     <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/>
>     <filter class="solr.LowerCaseFilterFactory"/>
>   </analyzer>
> </fieldType>
> {code}
> <charFilter/> can be multiple (chained). I'll post a JPEG file to show character
normalization sample soon.
> MOTIVATION:
> In Japan, there are two types of tokenizers -- N-gram (CJKTokenizer) and Morphological
Analyzer.
> When we use morphological analyzer, because the analyzer uses Japanese dictionary to
detect terms,
> we need to normalize characters before tokenization.
> I'll post a patch soon, too.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message