lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Beil <christian.a.b...@gmail.com>
Subject Re: Two-pass TokenFilter
Date Sun, 24 Aug 2014 22:25:35 GMT
Hi Ahmet,

Sorry, I wasn't very clear.
I need a TokenFilter that filters/skips some tokens, just like
FilteringTokenFilter.
To determine which tokens to filter, I first need to know all tokens.
Therefore I need to go through all tokens twice.

I implemented an TwoPassTokenFilter that is very similar to
CachingTokenFilter.
On the first call to incrementToken() it builds a cache and goes through al
tokens for the first pass.
The following calls to incrementToken() build the second pass.
In the second pass I can use information collected in the first pass.

Christian



2014-08-24 13:50 GMT+02:00 Ahmet Arslan <iorixxx@yahoo.com.invalid>:

> Hi,
>
> Can you elaborate more, what do you mean by "I need to know all tokens
> in advance."
>
> Ahmet
>
>
> On Wednesday, August 20, 2014 6:48 PM, Christian Beil <
> christian.a.beil@gmail.com> wrote:
> Hey guys,
>
> I need a TokenFilter that filters some tokens like the
> FilteringTokenFilter.
> The problem is, in order to do the filtering I need to know all tokens in
> advance.
>
> I thought I'll adapt the CachingTokenFilter in order to collect all tokens
> in the first pass.
> In the second pass it can use this information to filter the tokens.
>
> Or is there a better solution to do this?
>
> Thanks,
> Christian
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message