lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yonik Seeley (JIRA)" <>
Subject [jira] Commented: (LUCENE-1794) implement reusableTokenStream for all contrib analyzers
Date Tue, 11 Aug 2009 13:24:14 GMT


Yonik Seeley commented on LUCENE-1794:

It depends on the use case for CachingTokenFilter.
When it's used in places like QueryParser.getFieldQuery(), the consumer creates the CachingTokenFilter
and can rewind it too.

If one has managed to use the same instance more than once in the same document, other tricks
could be used such as resetting to the beginning after false is returned from incrementToken()
or implementing rewind in end().  Seems like either would work.

But in reality, the concept of CachingTokenFilter isn't really compatible with the concept
of reuse at all... so I don't think we necessarily need to do anything except document that
it's not reusable.  Adding rewind() to TokenStream won't solve this semantic problem.

> implement reusableTokenStream for all contrib analyzers
> -------------------------------------------------------
>                 Key: LUCENE-1794
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: contrib/analyzers
>            Reporter: Robert Muir
>            Assignee: Robert Muir
>            Priority: Minor
>             Fix For: 2.9
>         Attachments: LUCENE-1794.patch, LUCENE-1794.patch, LUCENE-1794.patch
> most contrib analyzers do not have an impl for reusableTokenStream
> regardless of how expensive the back compat reflection is for indexing speed, I think
we should do this to mitigate any performance costs. hey, overall it might even be an improvement!
> the back compat code for non-final analyzers is already in place so this is easy money
in my opinion.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message