lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Grant Ingersoll (JIRA)" <j...@apache.org>
Subject [jira] Resolved: (LUCENE-1406) new Arabic Analyzer (Apache license)
Date Tue, 21 Oct 2008 14:59:44 GMT

     [ https://issues.apache.org/jira/browse/LUCENE-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Grant Ingersoll resolved LUCENE-1406.
-------------------------------------

       Resolution: Fixed
    Fix Version/s: 2.9
    Lucene Fields: [Patch Available]  (was: [Patch Available, New])

> new Arabic Analyzer (Apache license)
> ------------------------------------
>
>                 Key: LUCENE-1406
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1406
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Analysis
>            Reporter: Robert Muir
>            Assignee: Grant Ingersoll
>            Priority: Minor
>             Fix For: 2.9
>
>         Attachments: LUCENE-1406.patch
>
>
> I've noticed there is no Arabic analyzer for Lucene, most likely because Tim Buckwalter's
morphological dictionary is GPL.
> However, it is not necessary  to have full morphological analysis engine for a quality
arabic search. 
> This implementation implements the light-8s algorithm present in the following paper:
http://ciir.cs.umass.edu/pubfiles/ir-249.pdf
> As you can see from the paper, improvement via this method over searching surface forms
(as lucene currently does) is significant, with almost 100% improvement in average precision.
> While I personally don't think all the choices were the best, and some easily improvements
are still possible, the major motivation for implementing it exactly the way it is presented
in the paper is that the algorithm is TREC-tested, so the precision/recall improvements to
lucene are already documented.
> For a stopword list, I used a list present at http://members.unine.ch/jacques.savoy/clef/index.html
simply because the creator of this list documents the data as BSD-licensed.
> This implementation (Analyzer) consists of above mentioned stopword list plus two filters:
>  ArabicNormalizationFilter: performs orthographic normalization (such as hamza seated
on alif, alif maksura, teh marbuta, removal of harakat, tatweel, etc)
>  ArabicStemFilter: performs arabic light stemming
> Both filters operate directly on termbuffer for maximum performance. There is no object
creation in this Analyzer.
> There are no external dependencies. I've indexed about half a billion words of arabic
text and tested against that.
> If there are any issues with this implementation I am willing to fix them. I use lucene
on a daily basis and would like to give something back. Thanks.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message