lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler (JIRA)" <>
Subject [jira] Commented: (LUCENE-1775) Change org.apache.lucene.analysis.shingle to use new TokenStream API
Date Sun, 02 Aug 2009 09:26:14 GMT


Uwe Schindler commented on LUCENE-1775:

This ShingleMatrixFilter is really a pain!

I think the ShingeMatrix is very "special" and only produce tokens with few correlation to
the original input stream, so it is not so bad, if the extra attributes get lost.

You could use a simple AttributeSource instead of EmptyTokenStream and create it with the
same AttributeFactory as the filter isself. Because of this, you could copyTo the extra Tokens
(currently implemented by the Token instance). This reuseableToken could also be an AttributeSource?
For me it is not really clear what all this copying between the attributes and the Token instance
does, but it seems that it could be converted to Attributes, too. If you do it that way, would
it be not work also with custom attributes? One possibility would be to copyTo the Tokens
around (or use States) and then modify the shingle speicfic things.

> Change org.apache.lucene.analysis.shingle to use new TokenStream API
> --------------------------------------------------------------------
>                 Key: LUCENE-1775
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Task
>          Components: contrib/analyzers
>            Reporter: Michael Busch
>            Assignee: Michael Busch
>            Priority: Minor
>             Fix For: 2.9
>         Attachments: lucene-1775.patch, lucene-1775.patch, lucene-1775.patch
> All other contrib streams/filters have already been converted with LUCENE-1460.
> The two shingle filters are the last ones we need to convert.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message