lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ahmet Arslan <iori...@yahoo.com.INVALID>
Subject Re: custom token filter generates empty tokens
Date Thu, 09 Oct 2014 16:34:18 GMT
Hi G.Long,

You can use TrimFilter+LengthFilter to remove empty/whitespace tokens.


Ahmet

On Thursday, October 9, 2014 5:54 PM, G.Long <jdevgl@gmail.com> wrote:
Hi :)

I wrote a custom token filter which removes special characters. 
Sometimes, all characters of the token are removed so the filter 
procudes an empty token. I would like to remove this token from the 
tokenstream but i'm not sure how to do that.

Is there something missing in my custom token filter or do I need to 
chain another custom token filter to remove empty tokens?

Regards :)

ps:

this is the code of my custom filter :

public class SpecialCharFilter extends TokenFilter {

     private final CharTermAttribute termAtt = 
addAttribute(CharTermAttribute.class);

     protected SpecialCharFilter(TokenStream input) {
         super(input);
     }

     @Override
     public boolean incrementToken() throws IOException {

         if (!input.incrementToken()) {
             return false;
         }

         final char[] buffer = termAtt.buffer();
         final int length = termAtt.length();
         final char[] newBuffer = new char[length];

         int newIndex = 0;
         for (int i = 0; i < length; i++) {
             if (!isFilteredChar(buffer[i])) {
                 newBuffer[newIndex] = buffer[i];
                 newIndex++;
             }
         }

         String term = new String(newBuffer);
         term = term.trim();
         char[] characters = term.toCharArray();
         termAtt.setEmpty();
         termAtt.copyBuffer(characters, 0, characters.length);

         return true;
     }
}

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message