lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shai Erera (JIRA)" <>
Subject [jira] Updated: (LUCENE-1068) Invalid behavior of StandardTokenizerImpl
Date Fri, 30 Nov 2007 06:53:43 GMT


Shai Erera updated LUCENE-1068:

    Attachment: StandardTokenizerImpl-3.patch

The previous patch I put was incorrect since it would still break existing applications. The
current patch does:
1. Introduces a new type ACRONYM_DEP which is deprecated and recognizes the old ACRONYM format.
2. Fixes ACRONYM to recognize LETTER + "." (LETTER + ".")+.
3. Added a public member to StandardTokenizer and StandardAnalyzer replaceDepAcronym which
can be set if the application would like the deprecated acronym format to be treated as ACRONYM
or HOST. The default behavior, if not set is to recognize the old ACRONYM as HOST.

This is how it should be used:
	public static void main(String[] args) throws Exception {
		parse("", false);
		parse("", true);
		parse("", true);
		parse("I.B.M.", true);

	public static void parse(String text, boolean replaceDepAcronym) throws Exception {
		StandardAnalyzer analyzer = new StandardAnalyzer();
    analyzer.replaceDepAcronym = replaceDepAcronym;
		TokenStream ts = analyzer.tokenStream("content", new StringReader(text));
		Token t;
		while ((t = != null) {
And here is the output:

The member is marked deprecated so we can remove it in the next release. Applications that
would like to new behavior need to do nothing, and therefore will not be impacted once we
remove that member. Applications that want the old behavior need to explicitly set it and
in the next major release remove it.

I think that solves it. How should I proceed?

> Invalid behavior of StandardTokenizerImpl
> -----------------------------------------
>                 Key: LUCENE-1068
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Analysis
>            Reporter: Shai Erera
>         Attachments: StandardTokenizerImpl-2.patch, StandardTokenizerImpl-3.patch, standardTokenizerImpl.jflex.patch,
> The following code prints the output of StandardAnalyzer:
>         Analyzer analyzer = new StandardAnalyzer();
>         TokenStream ts = analyzer.tokenStream("content", new StringReader("<some text>"));
>         Token t;
>         while ((t = != null) {
>             System.out.println(t);
>         }
> If you pass "", the output is (,0,11,type=<HOST>) (which
is correct in my opinion).
> However, if you pass "" (notice the extra '.' at the end), the output is
> I think the behavior in the second case is incorrect for several reasons:
> 1. It recognizes the string incorrectly (no argue on that).
> 2. It kind of prevents you from putting URLs at the end of a sentence, which is perfectly
> 3. An ACRONYM, at least to the best of my understanding, is of the form A.B.C. and not
> I looked at StandardTokenizerImpl.jflex and I think the problem comes from this definition:
> // acronyms: U.S.A., I.B.M., etc.
> // use a post-filter to remove dots
> ACRONYM    =  {ALPHA} "." ({ALPHA} ".")+
> Notice how the comment relates to acronym as U.S.A., I.B.M. and not something else. I
changed the definition to
> ACRONYM    =  {LETTER} "." ({LETTER} ".")+
> and it solved the problem.
> This was also reported here:

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message