lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Koji Sekiguchi (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (LUCENE-2207) CJKTokenizer generates tokens with incorrect offsets
Date Sun, 17 Jan 2010 04:55:55 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801312#action_12801312
] 

Koji Sekiguchi edited comment on LUCENE-2207 at 1/17/10 4:54 AM:
-----------------------------------------------------------------

Hi Robert, thank you for looking this so quickly!

{quote}
ok i found the bug. the problem is incrementToken() unconditionally increments the offset
before it starts its main loop:

line 165:

offset++;
{quote}

Indeed.

In attached patch, I added one more offset-- line and two more testcases. All tests pass and
this patch fixes the original problem that was found in Solr with FastVectorHighlighter.

      was (Author: koji):
    Hi Robert, thank you for looking this so quickly!

{quote}
ok i found the bug. the problem is incrementToken() unconditionally increments the offset
before it starts its main loop:

line 165:

offset++;
{quote}

Indeed.

In attached patch, I added one more offset-- line and two more testcases. All tests pass.
  
> CJKTokenizer generates tokens with incorrect offsets
> ----------------------------------------------------
>
>                 Key: LUCENE-2207
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2207
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: contrib/analyzers
>            Reporter: Koji Sekiguchi
>         Attachments: LUCENE-2207.patch, LUCENE-2207.patch, LUCENE-2207.patch, LUCENE-2207.patch,
TestCJKOffset.java
>
>
> If I index a Japanese *multi-valued* document with CJKTokenizer and highlight a term
with FastVectorHighlighter, the output snippets have incorrect highlighted string. I'll attach
a program that reproduces the problem soon.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message