lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Lassau (JIRA)" <>
Subject [jira] Updated: (LUCENE-1373) Most of the contributed Analyzers suffer from invalid recognition of acronyms.
Date Fri, 05 Sep 2008 08:29:44 GMT


Mark Lassau updated LUCENE-1373:

    Attachment: LUCENE-1373.patch

Added a draft patch to fix the default behaviour of StandardTokenizer.
This basically involved moving the logic of LUCENE-1151 from StandardAnalyzer to StandardTokenizer.

I added a unit test for StandardTokenizer, but unfortunately don't have time to add tests
for the language analyzers listed above (FrenchAnalyzer etc...).

I will be away for 3 weeks, so if anyone else wants to pick up this issue, that would be great
;) ... otherwise I will come back and look at it then.

> Most of the contributed Analyzers suffer from invalid recognition of acronyms.
> ------------------------------------------------------------------------------
>                 Key: LUCENE-1373
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Analysis, contrib/analyzers
>    Affects Versions: 2.3.2
>            Reporter: Mark Lassau
>            Priority: Minor
>         Attachments: LUCENE-1373.patch
> LUCENE-1068 describes a bug in StandardTokenizer whereby a string like ""
would be incorrectly tokenized as an acronym (note the dot at the end).
> Unfortunately, keeping the "backward compatibility" of a bug turns out to harm us.
> StandardTokenizer has a couple of ways to indicate "fix this bug", but unfortunately
the default behaviour is still to be buggy.
> Most of the non-English analyzers provided in lucene-analyzers utilize the StandardTokenizer,
and in v2.3.2 not one of these provides a way to get the non-buggy behaviour :(
> I refer to:
> * BrazilianAnalyzer
> * CzechAnalyzer
> * DutchAnalyzer
> * FrenchAnalyzer
> * GermanAnalyzer
> * GreekAnalyzer
> * ThaiAnalyzer

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message