lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Digy" <digyd...@gmail.com>
Subject RE: help customfilter with incrementToken() and AttributeSource APIs
Date Thu, 24 Dec 2009 18:15:27 GMT
The source code for LowerCaseFilter or StopFilter can be a good starting point.
DIGY

-----Original Message-----
From: maxSchlein [mailto:m_schlein@hotmail.com] 
Sent: Thursday, December 24, 2009 7:10 PM
To: java-user@lucene.apache.org
Subject: help customfilter with incrementToken() and AttributeSource APIs


In the current version of lucene, 3.0 the following methods are no longer
available.  
   - TokenStream.next() 
   - TokenStream.next(Token).  
   - Token.setTermText() 
   - Token.termText().  

The newer versions says to use, incrementToken() and AttributeSource APIs. 
But I cannot find much help using these in this way.  Any help again is
appreciated.  If anyone has a basic example, or can point me to something
useful that would be awesome.  Thanx.

Merry Christmas too.

public class CustomAnalyzer extends Analyzer
{
    @Override
    public TokenStream tokenStream(final String fieldName, final Reader
reader)
    {
        TokenStream ts = new WhitespaceTokenizer(reader);
        ts = new StopFilter(false, ts, StopAnalyzer.ENGLISH_STOP_WORDS_SET);
        ts = new LowerCaseFilter(ts);
        ts = new CustomFilter(ts);

        return ts;
    }

}

public class CustomFilter extends TokenFilter
{
    protected CustomFilter(TokenStream tokenStream)
    {
        super(tokenStream);
    }
    @Override
    public Token next(final Token reusableToken) throws IOException
    {
        Token nextToken = input.next(reusableToken);
        
        if(nextToken != null)
        {
           
nextToken.setTermText(nextToken.termText().replaceAll(":|,|\\(|\\)|“|~|;|&|\\.",""));
        }
        return nextToken;
    }
}
-- 
View this message in context: http://old.nabble.com/help-customfilter-with-incrementToken%28%29-and-AttributeSource-APIs-tp26915600p26915600.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message