lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler (JIRA)" <>
Subject [jira] Commented: (LUCENE-1919) Analysis back compat break
Date Fri, 18 Sep 2009 16:04:16 GMT


Uwe Schindler commented on LUCENE-1919:

Token.clone() is not called by next() anymore (and even not in 2.4.1) - because of this we
need the Payload cloning (Token.clone() would do it). The full private Token is done like
this now (and this was the same in 1057): next(Token) is simply called with a new Token instance.
The fix for this issue does it in the same way (like 2.4.1). The extension here is, that it
now works exactly the same with incrementToken(). After calling next(Token) with the private
token, the Payload is explicitely cloned.

As I told some comments above, I think more the problem is the following: If a next(Token)
or incrementToken() method sets a Payload, the payload data is not copied, its only a reference
to the byte array. E.g. the next(Token) methods uses a pre-allocated array and sets this always
as data (which is perfectly legal in the reuse case), only modifying the data contents. If
you call next(Token) with a new allocated Token, this Token is private, but if next(Token)
sets again the preallocated byte array, it is not private anymore (you will se the modifications
in the previous token, too). You would have the same bug like now (even in 2.4.1). I think
because of this the payload is cloned separately to be sure that it is private like the newly
allocated Token.

> Analysis back compat break
> --------------------------
>                 Key: LUCENE-1919
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Bug
>            Reporter: Yonik Seeley
>            Assignee: Uwe Schindler
>             Fix For: 2.9
>         Attachments: LUCENE-1919.patch, LUCENE-1919.patch, LUCENE-1919.patch, LUCENE-1919.patch,
LUCENE-1919.patch, LUCENE-1919.patch
> Old and new style token streams don't mix well.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message