lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler (JIRA)" <>
Subject [jira] Commented: (LUCENE-1843) Convert some tests to new TokenStream API, better support of cross-impl AttributeImpl.copyTo()
Date Sun, 23 Aug 2009 17:04:59 GMT


Uwe Schindler commented on LUCENE-1843:

>From a private mail with Robert Muir:

yes, all of what you mentioned are problems, and testing for
attributes that should be there is good in my opinion too.

I noticed the shingle problem as well, it was strange to test
termAtt.toString() and expect position increments or types to appear

one reason I asked about this, is at some point it would be nice to
try to factor test cases in lucene contrib. currently, they all have
same helper methods such as assertAnalyzesTo and this is silly in my

On Sun, Aug 23, 2009 at 12:57 PM, Uwe Schindler<> wrote:
> There are more problems. The test with getAttribute is good, if you are
> really sure, if the attribute is really available and want assert this. In
> all other cases addAttribute should be used to consume a TokenStream. The
> changed ones were problematic, because they used foreign TokenStreams, do
> not for sure have all these attributes.
> I thought, all tests in contrib use LuceneTestCase as superclass, but they
> use the standard junit class. Because of that I did not notice when I put
> setOnlyUseNewAPI(true) into LuceneTestCase.setUp(), that they are run with
> the default false setting.
> Other problems in these tests are, that some (shingle tests) use
> TermAttribute.toString() which looks different if the attribute is
> implemented by TermAttributeImpl (newAPI=true) or Token (newAPI=false).

> Convert some tests to new TokenStream API, better support of cross-impl AttributeImpl.copyTo()
> ----------------------------------------------------------------------------------------------
>                 Key: LUCENE-1843
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Analysis
>    Affects Versions: 2.9
>            Reporter: Uwe Schindler
>            Assignee: Uwe Schindler
>             Fix For: 2.9
>         Attachments: LUCENE-1843.patch
> This patch converts some remaining tests to the new TokenStream API and non-deprecated
> This patch also enhances AttributeImpl.copyTo() of Token and TokenWrapper to also support
copying e.g. TermAttributeImpl into Token. The target impl must only support all interfaces
but must not be of the same type. Token and TokenWrapper use optimized coping without casting
to 6 interfaces where possible.
> Maybe the special tokenizers in contrib (shingle matrix and so on using tokens to cache
may be enhanced by that). Also Yonik's request for optimized copying of states between incompatible
AttributeSources may be enhanced by that (possibly a new issue).

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message