lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Willnauer <>
Subject Re: How to add "tokens" to a stream
Date Mon, 07 Jun 2010 14:53:15 GMT
Hey there,
in lucene 3.0 / 2.9 the Token class has been remove / replaced with an
Attribute based API. A TokenStream operates on Attibutes it has
declared which are eventually accessed by the IndexWriter to create
the inverted index. There are Attributes like TermAttribute,
PositionIncrementAttribute and OffsetAttribute that roughly reflect
the fields of the Token class. If you want to return an extra token /
Term from the Attribute based API you just set the TermAttribute value
and all other attributes you wanna use and return true from the

something like this

public boolean incrementToken() {
     // process the tokens in the stream
    return true;
  } else if (!extraTokens.isEmtpy()){
    // return extra token
    return true;
  return false;


does that make sense?


On Mon, Jun 7, 2010 at 4:43 PM, Aad Nales <> wrote:
> Hi All,
> Years ago we implemented a Lucene solution which we are updating
> today, and i am a bit lost on the following.
> In Lucene 1.x and 2.x it was possible to add a token in a Filter
> simply by returning an extra Token when next was being called. What i
> can not find is an equivalent possiblity for adding a Token in a
> filter. The incrementToken() seem to be the logical place to implement
> similar functionality (at least it could trigger it) but i am a bit
> lost how to access the stream and insert an extra token. Any
> suggestions?
> cheers,
> Aad
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message