Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Date: Tue, 21 Jun 2011 08:21:48 +0000 (UTC)
From: "Dawid Weiss (JIRA)" <jira@apache.org>
To: dev@lucene.apache.org
Message-ID: 
 <391329183.23703.1308644508967.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (LUCENE-2341) explore morfologik integration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/LUCENE-2341?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D130=
52421#comment-13052421 ]=20

Dawid Weiss commented on LUCENE-2341:
-------------------------------------

I did some analyses on both dictionaries.
{noformat}
Number of lines (distict surface forms):

  3.662.366 morfologik.utf8
  5.086.141 sgjp.utf8

Distinct words (not in both):

  2.729.334 unique.utf8

  - upper/lower case (morfologik has upper case forms, morfeusz only lower =
case surface forms)
   =20
    acerze
    Acerze

  - very rare or jargon;

    abszminka
    abszytowa=C5=82em
    acetobakteria
    acetarsolowi
    niebombiasto
    hakatystce
    hakatystyczno=C5=9Bciach
    warz=C5=BCe

  - differences in spelling;

    abelard
    ab=C3=A9lard

  - acronyms and super-short stuff

    aap
    aar

Dictinct normalized (lowercase):

  2.564.366 lowered.utf8

  Most of these are very infrequent words or inflection forms. There are mi=
nor differences or
  missing surface forms in both dictionaries, as in here (mz - morfeusz, mk=
 - morfologik):

mz> hakersko
mz> hakersko=C5=9B=C4=87
mz> hakersko=C5=9Bci
mz> hakersko=C5=9Bci=C4=85
mz> hakersko=C5=9Bciach
mz> hakersko=C5=9Bciami
mz> hakersko=C5=9Bciom
mk> hakerstw
mk> hakerstwa
...
mk> hakowa=C5=82y=C5=9Bmy
mk> hakowa=C5=84
mk> hakowaniach
mk> hakowaniami
mk> hakowaniom
mz> hakowato=C5=9B=C4=87
mz> hakowato=C5=9Bci
mz> hakowato=C5=9Bci=C4=85
mz> hakowato=C5=9Bciach
mz> hakowato=C5=9Bciami
mz> hakowato=C5=9Bciom
{noformat}

So... the conclusion is pretty consistent with Zipf's law: both dictionarie=
s have a fairly different coverage, even if they're quite large. We don't h=
ave a frequency dictionary for Polish, but I assume most of these surface f=
orms are purely theoretical and occur super-rarely in practice. This said, =
I think we should use BOTH dictionaries -- after all there's no harm done i=
f we overdo the lemmatization process a little bit, is there?

So... my proposal would be this: I'll integrate Morfeusz's dictionary in Mo=
rfologik (as an alternative dictionary one can load and use).=20

Eventually it would be probably sensible to limit the automaton for use in =
Lucene to store surface forms and lemmas only (no POS tags) and merge both =
dictionaries into a single automaton... but this can  be a future improveme=
nt.


> explore morfologik integration
> ------------------------------
>
>                 Key: LUCENE-2341
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2341
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/analysis
>            Reporter: Robert Muir
>            Assignee: Dawid Weiss
>         Attachments: LUCENE-2341.diff, morfologik-stemming-1.5.0.jar
>
>
> Dawid Weiss mentioned on LUCENE-2298 that there is another Polish stemmer=
 available:
> http://sourceforge.net/projects/morfologik/
> This works differently than LUCENE-2298, and ideally would be another opt=
ion for users.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org