lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: I need to implement a TokenFilter to break season07
Date Tue, 15 Dec 2009 12:47:30 GMT
> > And if you do it yourself, don't forget to call clearAttributes()
> whenever
> > you produce new tokens (else you may have bugs in the token increments).
> In
> > the old token api its Token.clear()... Just a warning!
> 
> This comment has worried me, is this ok or am i meant to call
> clearAttributes() somewhere

Your filter is fine. As noted, you should call clearAttributes, when you
produce new tokens, but you are only modifying existing ones. The example
Weiwei was mendtioning was to split a Token into two. So for the second
generated token you must really initialize all attributes to default values.

This is why the warning.


> public class StripLeadingZeroFilter extends TokenFilter {
>     /**
>      * Construct filtering <i>in</i>.
>      */
>     public StripLeadingZeroFilter(TokenStream in) {
>         super(in);
>         termAtt = (TermAttribute) addAttribute(TermAttribute.class);
>     }
> 
>     private TermAttribute termAtt;
> 
>     /**
>      *
>      * <p>Removes zeroes if first char in token
>      */
>     public final boolean incrementToken() throws java.io.IOException {
>         if (!input.incrementToken()) {
>             return false;
>         }
> 
>         char[] buffer = termAtt.termBuffer();
>         final int bufferLength = termAtt.termLength();
> 
>         if (buffer[0] == '0') {
>             for (int i = 1; i < bufferLength; i++) {
>                 char c = buffer[i];
>                 buffer[i - 1] = c;
>             }
>             termAtt.setTermLength(bufferLength - 1);
>             return true;
>         } else {
>             return true;
>         }
>     }
> 
> }
> 
> 
> thanks Paul
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message