lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <cutt...@lucene.com>
Subject Re: implementing a TokenFilter for aliases
Date Fri, 05 Dec 2003 18:47:40 GMT
Position increments are for relative token positions.  A position 
increment of zero means that a token is logically at the same position 
as the previous token.  A position increment of one means that a token 
immediately follows the preceding token in the stream, it's the next 
token to the right (in a left-to-right language).  A position increment 
of two means that it is two tokens past the previous token, that there's 
a "phantom" token between them, inhibiting exact phrase matches.

You're setting the position increment to things based on the number of 
characters in the token's text.  That makes no sense.  Token positions 
are not character positions.  I think what you want to do is use a 
positionIncrement of zero, so the tokens lie at the same position.

Doug

Allen Atamer wrote:
> The FAQ describes implementing a TokenFilter for applying aliases. I have a
> trouble accomplishing this.
>  
> This is the code that I have so far for the next Method within AliasFilter.
> After reading some posts, I also got the idea to call
> setPositionIncrement(). Neither way works, because when I search for the
> alias, no search results come back.
>  
> Thank you for your help,
>  
> Allen Atamer
>  
> ----
>  
>   public Token next() throws java.io.IOException {
>     Token token = tokenStream.next();
>  
>     if (aliasMap == null || token == null) {
>       return token;
>     }
>  
>     TermData t = (TermData)aliasMap.get(token.termText());
>  
>     if (t == null) {
>       return token;
>     }
>  
>     String tokenText = AliasManager.replaceIgnoreCase(
>         token.termText(), t.getTerm(), t.getTeach());
>  
>     int increment = tokenText.length() - token.termText().length();
>     if (increment > 0) {
>       token.setPositionIncrement(increment);
>     }
>  
>     return new Token(tokenText, token.startOffset(), token.endOffset());
>   }
>  
>  
>  
>  
>  
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message