Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Message-ID: <1526869688.1242049665643.JavaMail.jira@brutus>
Date: Mon, 11 May 2009 06:47:45 -0700 (PDT)
From: "Michael McCandless (JIRA)" <jira@apache.org>
To: java-dev@lucene.apache.org
Subject: [jira] Commented: (LUCENE-1629) contrib intelligent Analyzer for
 Chinese
In-Reply-To: <445685380.1241412690530.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/LUCENE-1629?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D127=
08032#action_12708032 ]=20

Michael McCandless commented on LUCENE-1629:
--------------------------------------------

I do have the file, but at runtime the JRE cannot locate it using Class.get=
ResourceAsStream().

Are you able to run "ant test -Dtestcase=3DTestSmartChineseAnalyzer" from t=
he command line in contrib/analzyers successfully?

> contrib intelligent Analyzer for Chinese
> ----------------------------------------
>
>                 Key: LUCENE-1629
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1629
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: contrib/analyzers
>    Affects Versions: 2.4.1
>         Environment: for java 1.5 or higher, lucene 2.4.1
>            Reporter: Xiaoping Gao
>            Assignee: Michael McCandless
>             Fix For: 2.9
>
>         Attachments: analysis-data.zip, bigramdict.mem, coredict.mem, LUC=
ENE-1629-java1.4.patch
>
>
> I wrote a Analyzer for apache lucene for analyzing sentences in Chinese l=
anguage. it's called "imdict-chinese-analyzer", the project on google code =
is here: http://code.google.com/p/imdict-chinese-analyzer/
> In Chinese, "=E6=88=91=E6=98=AF=E4=B8=AD=E5=9B=BD=E4=BA=BA"(I am Chinese)=
, should be tokenized as "=E6=88=91"(I)   "=E6=98=AF"(am)   "=E4=B8=AD=E5=
=9B=BD=E4=BA=BA"(Chinese), not "=E6=88=91" "=E6=98=AF=E4=B8=AD" "=E5=9B=BD=
=E4=BA=BA". So the analyzer must handle each sentence properly, or there wi=
ll be mis-understandings everywhere in the index constructed by Lucene, and=
 the accuracy of the search engine will be affected seriously!
> Although there are two analyzer packages in apache repository which can h=
andle Chinese: ChineseAnalyzer and CJKAnalyzer, they take each character or=
 every two adjoining characters as a single word, this is obviously not tru=
e in reality, also this strategy will increase the index size and hurt the =
performance baddly.
> The algorithm of imdict-chinese-analyzer is based on Hidden Markov Model =
(HMM), so it can tokenize chinese sentence in a really intelligent way. Tok=
enizaion accuracy of this model is above 90% according to the paper "HHMM-b=
ased Chinese Lexical analyzer ICTCLAL" while other analyzer's is about 60%.
> As imdict-chinese-analyzer is a really fast and intelligent. I want to co=
ntribute it to the apache lucene repository.

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org