lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Muir (JIRA)" <>
Subject [jira] Commented: (LUCENE-2167) StandardTokenizer Javadoc does not correctly describe tokenization around punctuation characters
Date Wed, 24 Feb 2010 23:07:27 GMT


Robert Muir commented on LUCENE-2167:

bq. I'll take a crack at understanding unicode standard tokenization, as you'd suggested originally,
and try and produce something as soon as I get a chance.

I would love it if you could produce a grammar that implemented UAX#29!

If so, in my opinion it should become the StandardAnalyzer for the next lucene version. If
I thought I could do it correctly, I would have already done it, as the support for the unicode
properties needed to do this is now in the trunk of Jflex!

here are some references that might help: 
The standard itself:

particularly the "Testing" portion:

Unicode provides a WordBreakTest.txt file, that we could use from Junit, to help verify correctness:

I'll warn you I think it might be hard, but perhaps its not that bad. In particular the standard
is defined in terms of "chained" rules, and Jflex doesnt support rule chaining, but I am not
convinced we need rule chaining to implement WordBreak (maybe LineBreak, but maybe WordBreak
can be done easily without it?) 

Steven Rowe is the expert on this stuff, maybe he has some ideas.

> StandardTokenizer Javadoc does not correctly describe tokenization around punctuation
> ------------------------------------------------------------------------------------------------
>                 Key: LUCENE-2167
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Bug
>    Affects Versions: 2.4.1, 2.9, 2.9.1, 3.0
>            Reporter: Shyamal Prasad
>            Priority: Minor
>         Attachments: LUCENE-2167.patch, LUCENE-2167.patch
>   Original Estimate: 0.5h
>  Remaining Estimate: 0.5h
> The Javadoc for StandardTokenizer states:
> {quote}
> Splits words at punctuation characters, removing punctuation. 
> However, a dot that's not followed by whitespace is considered part of a token.
> Splits words at hyphens, unless there's a number in the token, in which case the whole

> token is interpreted as a product number and is not split.
> {quote}
> This is not accurate. The actual JFlex implementation treats hyphens interchangeably
> punctuation. So, for example "video,mp4,test" results in a *single* token and not three
> as the documentation would suggest.
> Additionally, the documentation suggests that "video-mp4-test-again" would become a single
> token, but in reality it results in two tokens: "video-mp4-test" and "again".
> IMHO the parser implementation is fine as is since it is hard to keep everyone happy,
but it is probably
> worth cleaning up the documentation string. 
> The patch included here updates the documentation string and adds a few test cases to
confirm the cases described above.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message