lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Muir (JIRA)" <>
Subject [jira] Resolved: (LUCENE-2219) improve BaseTokenStreamTestCase to test end()
Date Sun, 17 Jan 2010 21:46:54 GMT


Robert Muir resolved LUCENE-2219.

    Resolution: Fixed

> improve BaseTokenStreamTestCase to test end()
> ---------------------------------------------
>                 Key: LUCENE-2219
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Analysis, contrib/analyzers
>    Affects Versions: 2.9, 3.0
>            Reporter: Robert Muir
>            Assignee: Robert Muir
>             Fix For: 2.9.2, 3.0.1, 3.1
>         Attachments: LUCENE-2219.patch, LUCENE-2219.patch, LUCENE-2219.patch
> If offsetAtt/end() is not implemented correctly, then there can be problems with highlighting:
see LUCENE-2207 for an example with CJKTokenizer.
> In my opinion you currently have to write too much code to test this.
> This patch does the following:
> * adds optional Integer finalOffset (can be null for no checking) to assertTokenStreamContents
> * in assertAnalyzesTo, automatically fill this with the String length()
> In my opinion this is correct, for assertTokenStreamContents the behavior should be optional,
it may not even have a Tokenizer. If you are using assertTokenStreamContents with a Tokenizer
then simply provide the extra expected value to check it.
> for assertAnalyzesTo then it is implied there is a tokenizer so it should be checked.
> the tests pass for core but there are failures in contrib even besides CJKTokenizer (apply
Koji's patch from LUCENE-2207, it is correct). Specifically ChineseTokenizer has a similar

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message