Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Message-ID: <7832941.148651271969691130.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 16:54:51 -0400 (EDT)
From: "Robert Muir (JIRA)" <jira@apache.org>
To: java-dev@lucene.apache.org
Subject: [jira] Updated: (LUCENE-2414) add icu-based tokenizer for unicode
 text segmentation
In-Reply-To: <17205450.148581271969572970.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/LUCENE-2414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Muir updated LUCENE-2414:
--------------------------------

    Attachment: LUCENE-2414.patch

attached is a patch.

after applying it run 'ant genrbbi', which compiles the rule tailorings to binary DFAs for faster loading.
you dont have to do this, e.g. if you want you can always create these from String, but this is much faster.


> add icu-based tokenizer for unicode text segmentation
> -----------------------------------------------------
>
>                 Key: LUCENE-2414
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2414
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: contrib/*
>    Affects Versions: 3.1
>            Reporter: Robert Muir
>             Fix For: 3.1
>
>         Attachments: LUCENE-2414.patch
>
>
> I pulled out the last part of LUCENE-1488, the tokenizer itself and cleaned it up some.
> The idea is simple:
> * First step is to divide text into writing system boundaries (scripts)
> * You supply an ICUTokenizerConfig (or just use the default) which lets you tailor segmentation on a per-writing system basis.
> * This tailoring can be any BreakIterator, so rule-based or dictionary-based or your own.
> The default implementation (if you do not customize) is just to do UAX#29, but with tailorings for stuff with no clear word division:
> * Thai (uses dictionary-based word breaking)
> * Khmer, Myanmar, Lao (uses custom rules for syllabification)
> Additionally as more of an example i have a tailoring for hebrew that treats the punctuation special. (People have asked before
> for ways to make standardanalyzer treat dashes differently, etc)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org