lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Keegan" <>
Subject Re: multiple tokens from a single input token
Date Mon, 10 Nov 2003 15:34:56 GMT
Thanks for the clarification. I thought there might have been a way to do it
without a stack and without keeping any state information. The AliasFilter
should accomplish the task.


----- Original Message ----- 
From: "Pierrick Brihaye" <>
To: "Lucene Users List" <>
Sent: Monday, November 10, 2003 10:14 AM
Subject: Re: multiple tokens from a single input token


MOYSE Gilles (Cetelem) a écrit:

> I experienced the same problem, and I used the following solution (maybe
> the good one, but it works, and not too slowly).
> The problem was to detect synonyms. I used a synonyms file, made up of
> kind of lines :
> a b c
> d e f

Mmmh... 1 for 1. The question was deliberatly a 1 to N tokenization.

> I used a FIFO stack to solve that.

Yes : the "token stack" does the trick. My code was actually a token
stack but... less beautiful (and more generic) than the code provided
just a bit later :-)

> When the filter receives a token, it checks whether the stack is empty or
> not. If it is, then it returns the received token. If it is not empty,
> it returns the poped (i.e. the first which was pushed. It's better to use
> FIFO stack to keep a correct order) value from the stack.
> When you receive the 'null' token, indicating the end of stream, then you
> continue returning the poped values from yoour stack until it is empty.
> you return 'null'.

That's it.

Please do notice that the stack is necessarily declared outside of the
next() method, i.e. it is an global instance variable. Maybe Peter
Keegan missed this point ?


Pierrick Brihaye, informaticien
Service régional de l'Inventaire
DRAC Bretagne

To unsubscribe, e-mail:
For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message