lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steven Rowe (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (LUCENE-2167) StandardTokenizer Javadoc does not correctly describe tokenization around punctuation characters
Date Wed, 24 Feb 2010 23:29:27 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-2167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12838081#action_12838081
] 

Steven Rowe edited comment on LUCENE-2167 at 2/24/10 11:27 PM:
---------------------------------------------------------------

I wrote word break rules grammar specifications for JFlex 1.5.0-SNAPSHOT and both Unicode
versions 5.1 and 5.2 - you can see the files here:

http://jflex.svn.sourceforge.net/viewvc/jflex/trunk/testsuite/testcases/src/test/cases/unicode-word-break/

The files are {{UnicodeWordBreakRules_5_\*.\*}} - these are written to: parse the Unicode
test files; run the generated scanner against each composed test string; output the break
opportunities/prohibitions in the same format as the test files; and then finally compare
the output against the test file itself, looking for a match.  (These tests currently pass.)

The .flex files would need to be significantly changed to be used as a StandardTokenizer replacement,
but you can get an idea from them how to implement the Unicode word break rules in (as yet
unreleased version 1.5.0) JFlex syntax.

      was (Author: steve_rowe):
    I wrote word break rules grammar specifications for JFlex 1.5.0-SNAPSHOT and both Unicode
versions 5.1 and 5.2 - you can see the files here:

http://jflex.svn.sourceforge.net/viewvc/jflex/trunk/testsuite/testcases/src/test/cases/unicode-word-break/

The files are UnicodeWordBreakRules_5_*.* - these are written to: parse the Unicode test files;
run the generated scanner against each composed test string; output the break opportunities/prohibitions
in the same format as the test files; and then finally compare the output against the test
file itself, looking for a match.  (These tests currently pass.)

The .flex files would need to be significantly changed to be used as a StandardTokenizer replacement,
but you can get an idea from them how to implement the Unicode word break rules in (as yet
unreleased version 1.5.0) JFlex syntax.
  
> StandardTokenizer Javadoc does not correctly describe tokenization around punctuation
characters
> ------------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-2167
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2167
>             Project: Lucene - Java
>          Issue Type: Bug
>    Affects Versions: 2.4.1, 2.9, 2.9.1, 3.0
>            Reporter: Shyamal Prasad
>            Priority: Minor
>         Attachments: LUCENE-2167.patch, LUCENE-2167.patch
>
>   Original Estimate: 0.5h
>  Remaining Estimate: 0.5h
>
> The Javadoc for StandardTokenizer states:
> {quote}
> Splits words at punctuation characters, removing punctuation. 
> However, a dot that's not followed by whitespace is considered part of a token.
> Splits words at hyphens, unless there's a number in the token, in which case the whole

> token is interpreted as a product number and is not split.
> {quote}
> This is not accurate. The actual JFlex implementation treats hyphens interchangeably
with
> punctuation. So, for example "video,mp4,test" results in a *single* token and not three
tokens
> as the documentation would suggest.
> Additionally, the documentation suggests that "video-mp4-test-again" would become a single
> token, but in reality it results in two tokens: "video-mp4-test" and "again".
> IMHO the parser implementation is fine as is since it is hard to keep everyone happy,
but it is probably
> worth cleaning up the documentation string. 
> The patch included here updates the documentation string and adds a few test cases to
confirm the cases described above.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message