commons-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Neidhart (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CODEC-187) Beider Morse Phonetic Matching producing incorrect tokens
Date Fri, 04 Jul 2014 13:51:34 GMT

    [ https://issues.apache.org/jira/browse/CODEC-187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14052446#comment-14052446
] 

Thomas Neidhart edited comment on CODEC-187 at 7/4/14 1:51 PM:
---------------------------------------------------------------

The latest attached patch (sync_with_v3.3) applies the following changes:

 * complete sync with rules from original Beider/Morse v3.3
 * use different language detection for each name type (ash, gen, sep) previously only one
generic rule set was used, but the original code distinguishes between the name types
 * fix a bug when applying the final rules: if two phonemes with the same text but different
language sets were encountered, only the first one was stored as the comparator did not take
the language into account

The results are identical for the previously reported failures.

One exception is the case when explicitly selecting hebrew as language, then the results are
different, but I do not understand the original code in this regard. We may need to contact
the authors about this issue.


was (Author: tn):
The latest attached patch (sync_with_v3.3) applies the following changes:

 * complete sync with rules from original Beider/Morse v3.3
 * use different language detection for each name type (ash, gen, sep) previously only one
generic rule set was used, but the original code distinguishes between the name types
 * fix a bug when applying the final rules: if two phonemes with the same text but different
language sets where encountered, only the first one was stored as the comparator did not take
the language into account

The results are identical for the previously reported failures.

One exception is the case when explicitly selecting hebrew as language, then the results are
different, but I do not understand the original code in this regard. We may need to contact
the authors about this issue.

> Beider Morse Phonetic Matching producing incorrect tokens
> ---------------------------------------------------------
>
>                 Key: CODEC-187
>                 URL: https://issues.apache.org/jira/browse/CODEC-187
>             Project: Commons Codec
>          Issue Type: Bug
>    Affects Versions: 1.9
>            Reporter: michael tobias
>            Priority: Minor
>             Fix For: 1.10
>
>         Attachments: CODEC-187.patch, CODEC-187_ashkenazi_approx_any.patch, CODEC-187_ashkenazi_approx_any_v2.patch,
CODEC_187_sync_with_v3.3.diff
>
>
> I believe the Beider Morse Phonetic Matching algorithm was added in Commons Codec 1.6
> The BMPM algorithm is an EVOLVING algorithm that is currently on version 3.02 though
it had been static since version 3.01 dated 19 Dec 2011 (it was first available as opensource
as version 1.00 on 6 May 2009).
> I can see nothing in the Commons Codec Docs to say which version of BMPM was implemented
so I am not sure if the problem with the algorithm as coded in the Codec is simply an old
version or whether there are more basic problems with the implementation.
> How do I determine the version of the algorithm that was implemented in the Commons Codec?
> How do we ensure that the algorithm is updated if/when the BMPM algorithm changes?
> How do we ensure that the algorithm as coded in the Commons Codec is accurate and working
as expected?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message