lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steven A Rowe <>
Subject RE: Access next token in a stream
Date Thu, 09 Feb 2012 19:54:10 GMT
Hi Damerian,

One way to handle your scenario is to hold on to the previous token, and only emit a token
after you reach at least the second token (or at end-of-stream).  Your incrementToken() method
could look something like:

1. Get current attributes: input.incrementToken()
2. If previous token does not exist:
      2a. Store current attributes as previous token (see AttributeSource#cloneAttributes)
	2b. Get current attributes: input.incrementToken()
3. Check for & store conditions that will affect previous token's attributes
4. Store current attributes as next token (see AttributeSource#cloneAttributes)
5. Copy previous token into current attributes (see AttributeSource#copyTo);
   the target will be "this", which is an AttributeSource.
6. Make changes based on conditions found in step #3 above
7. set previous token = next token
8. return true

(Everywhere I say "token" I mean "instance of AttributeSource".)

The final token in the input stream will need special handling, as will single-token input

Good luck,

> -----Original Message-----
> From: Damerian []
> Sent: Thursday, February 09, 2012 2:19 PM
> To:
> Subject: Access next token in a stream
> Hello i want to implement my custom filter, my wuestion is quite simple
> but i cannot find a solution to it no matter how i try:
> How can i access the TermAttribute of the  next token than the one i
> currently have in my stream?
> For example in  the phrase "My name is James Bond" if let's say i am in
> the token [My], i would like to be able to check the TermAttribute of
> the following token [name] and fix my position increment accordingly.
> Thank you in advance!
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

View raw message