lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Beil <>
Subject Re: Two-pass TokenFilter
Date Sun, 24 Aug 2014 22:25:35 GMT
Hi Ahmet,

Sorry, I wasn't very clear.
I need a TokenFilter that filters/skips some tokens, just like
To determine which tokens to filter, I first need to know all tokens.
Therefore I need to go through all tokens twice.

I implemented an TwoPassTokenFilter that is very similar to
On the first call to incrementToken() it builds a cache and goes through al
tokens for the first pass.
The following calls to incrementToken() build the second pass.
In the second pass I can use information collected in the first pass.


2014-08-24 13:50 GMT+02:00 Ahmet Arslan <>:

> Hi,
> Can you elaborate more, what do you mean by "I need to know all tokens
> in advance."
> Ahmet
> On Wednesday, August 20, 2014 6:48 PM, Christian Beil <
>> wrote:
> Hey guys,
> I need a TokenFilter that filters some tokens like the
> FilteringTokenFilter.
> The problem is, in order to do the filtering I need to know all tokens in
> advance.
> I thought I'll adapt the CachingTokenFilter in order to collect all tokens
> in the first pass.
> In the second pass it can use this information to filter the tokens.
> Or is there a better solution to do this?
> Thanks,
> Christian
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message