lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Muir <>
Subject Re: Issue with Solr TokenFilter and the new TokenStream API
Date Thu, 06 Aug 2009 14:58:03 GMT
Mark, I looked at this and think it might be unrelated to tokenstreams.

I think the length argument being provided to processWord(char[]
buffer, int offset, int length, int wordCount) in that filter might be
incorrectly calculated.
This is the method that checks the keep list.

(There is trailing trash on the end of tokens, even with the previous
version of lucene in Solr).
It just so happens the tokens with trailing trash were ones that were
keep words in the previous version, so the test didnt fail.

different tokens have trailing trash in the current version
(specifically some of the "the" tokens), so its failing now.

On Thu, Aug 6, 2009 at 10:14 AM, Mark Miller<> wrote:
> I think there is an issue here, but I didn't follow the TokenStream
> improvements very closely.
> In Solr, CapitalizationFilterFactory has a CharArray set that it loads up
> with keep words - it then checks (with the old TokenStream API) each token
> (char array) to see if it should keep it. I think because of the cloning
> going on in next, this breaks and you can't match anything in the keep set.
> Does that make sense?
> --
> - Mark
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

Robert Muir

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message