mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dawid Weiss <dawid.we...@cs.put.poznan.pl>
Subject Re: Tags generation?
Date Fri, 03 Aug 2012 20:08:19 GMT
I know, I know. :) Just wanted to mention that it could lead to funny
results, that's all. There are lots of way of doing proper form
disambiguation, including shallow tagging which then allows to
retrieve correct base forms for lemmas, not stems. Stemming is
typically good enough (and fast) so your advise was 100% fine.

Dawid

On Fri, Aug 3, 2012 at 9:31 PM, Ted Dunning <ted.dunning@gmail.com> wrote:
> This is definitely just the first step.  Similar goofs happen with
> inappropriate stemming.  For instance, AIDS should not stem to aid.
>
> A reasonable way to find and classify exceptional cases is to look at
> cooccurrence statistics.  The contexts of original forms can be examined to
> find cases where there is a clear semantic mismatch between the original
> and the set of all forms that stem to the same form.
>
> But just picking the most common that is present in the document is a
> pretty good step for all that it produces some oddities.  The results are
> much better than showing a user the stemmed forms.
>
> On Fri, Aug 3, 2012 at 1:05 PM, Dawid Weiss <dawid.weiss@cs.put.poznan.pl>wrote:
>
>> > Unstemming is pretty simple.  Just build an unstemming dictionary based
>> on
>> > seeing what word forms have lead to a stemmed form.  Include frequencies.
>>
>> This can lead to very funny (or not, depends how you look at it)
>> mistakes when different lemmas stem to the same token. How frequent
>> and important this phenomenon is varies from language to language (and
>> can be calculated apriori).
>>
>> Dawid
>>

Mime
View raw message