lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Rowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-5447) StandardTokenizer should break at consecutive chars matching Word_Break = MidLetter, MidNum and/or MidNumLet
Date Wed, 14 May 2014 23:10:36 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-5447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998249#comment-13998249
] 

Steve Rowe commented on LUCENE-5447:
------------------------------------

{quote}
bq. I converted the tests to the WordBreakTest.txt format and submitted them (along with an
explanation pointing to this issue) through the Unicode.org contact form at http://www.unicode.org/reporting.html.

The message I sent is now recorded as the second email in the feedback for Proposed Update
UAX #29, Unicode Text Segmentation: http://www.unicode.org/review/pri265/
{quote}

The Unicode Technical Committee emailed me today to tell me that they would be adding test
cases for this problem to Unicode 8.0, but not to the upcoming 7.0 release.

> StandardTokenizer should break at consecutive chars matching Word_Break = MidLetter,
MidNum and/or MidNumLet
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-5447
>                 URL: https://issues.apache.org/jira/browse/LUCENE-5447
>             Project: Lucene - Core
>          Issue Type: Bug
>          Components: modules/analysis
>    Affects Versions: 4.6.1
>            Reporter: Steve Rowe
>            Assignee: Steve Rowe
>             Fix For: 4.7, 5.0
>
>         Attachments: LUCENE-5447-take2.patch, LUCENE-5447-test.patch, LUCENE-5447.patch,
LUCENE-5447.patch
>
>
> StandardTokenizer should split all of the following sequences into two tokens each, but
they are all instead kept intact and output as single tokens:
> {noformat}
> "A::B"           (':' is in \p{Word_Break = MidLetter})
> "1..2", "A..B"   ('.' is in \p{Word_Break = MidNumLet})
> "A.:B"
> "A:.B"
> "1,,2"           (',' is in \p{Word_Break = MidNum})
> "1,.2"
> "1.,2"
> {noformat}
> Unfortunately, the word break test data released with Unicode, e.g. for Unicode 6.3 [http://www.unicode.org/Public/6.3.0/ucd/auxiliary/WordBreakTest.txt],
and incorporated into a versioned Lucene test, e.g. {{WordBreakTestUnicode_6_3_0}}, doesn't
cover these cases.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message