lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aad Nales <aad.na...@gmail.com>
Subject Re: How to add "tokens" to a stream
Date Mon, 07 Jun 2010 14:55:56 GMT
Yes it does,

The key bit is the part with the termAttribute...

thanks a lot,

cheers,
Aad

On Mon, Jun 7, 2010 at 4:53 PM, Simon Willnauer
<simon.willnauer@googlemail.com> wrote:
> Hey there,
> in lucene 3.0 / 2.9 the Token class has been remove / replaced with an
> Attribute based API. A TokenStream operates on Attibutes it has
> declared which are eventually accessed by the IndexWriter to create
> the inverted index. There are Attributes like TermAttribute,
> PositionIncrementAttribute and OffsetAttribute that roughly reflect
> the fields of the Token class. If you want to return an extra token /
> Term from the Attribute based API you just set the TermAttribute value
> and all other attributes you wanna use and return true from the
> TokenStream#incrementToken()
>
> something like this
>
> public boolean incrementToken() {
>  if(input.incrementToken()){
>     // process the tokens in the stream
>    return true;
>  } else if (!extraTokens.isEmtpy()){
>    // return extra token
>    termAttribute.setTerm(extraTokens.pop())
>    return true;
>  }
>  return false;
>
> }
>
> does that make sense?
>
> simon
>
> On Mon, Jun 7, 2010 at 4:43 PM, Aad Nales <aad.nales@gmail.com> wrote:
>> Hi All,
>>
>> Years ago we implemented a Lucene solution which we are updating
>> today, and i am a bit lost on the following.
>>
>> In Lucene 1.x and 2.x it was possible to add a token in a Filter
>> simply by returning an extra Token when next was being called. What i
>> can not find is an equivalent possiblity for adding a Token in a
>> filter. The incrementToken() seem to be the logical place to implement
>> similar functionality (at least it could trigger it) but i am a bit
>> lost how to access the stream and insert an extra token. Any
>> suggestions?
>>
>> cheers,
>> Aad
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message