lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shyamal Prasad (JIRA)" <j...@apache.org>
Subject [jira] Created: (LUCENE-2167) StandardTokenizer Javadoc does not correctly describe tokenization around punctuation characters
Date Wed, 16 Dec 2009 18:50:18 GMT
StandardTokenizer Javadoc does not correctly describe tokenization around punctuation characters
------------------------------------------------------------------------------------------------

                 Key: LUCENE-2167
                 URL: https://issues.apache.org/jira/browse/LUCENE-2167
             Project: Lucene - Java
          Issue Type: Bug
    Affects Versions: 3.0, 2.9.1, 2.9, 2.4.1
            Reporter: Shyamal Prasad
            Priority: Minor


The Javadoc for StandardTokenization states:

{bq}
Splits words at punctuation characters, removing punctuation. 
However, a dot that's not followed by whitespace is considered part of a token.

Splits words at hyphens, unless there's a number in the token, in which case the whole 
token is interpreted as a product number and is not split.
{bq}

This is not accurate. The actual JFlex implementation treats hyphens interchangeably with
punctuation. So, for example "video,mp4,test" results in a *single* token and not three tokens
as the documentation would suggest.

Additionally, the documentation suggests that "video-mp4-test-again" would become a single
token, but in reality it results in two tokens: "video-mp4-test" and "again".

IMHO the parser implementation is fine as is since it is hard to keep everyone happy, but
it is probably
worth cleaning up the documentation string.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message