lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Sokolov <soko...@ifactory.com>
Subject Re: adding attributes to TokenStream
Date Mon, 31 Dec 2012 20:57:40 GMT
On 12/31/2012 11:39 AM, Itai Peleg wrote:
> Hi all,
>
> Can someone please post a simple example showing how to add additional
> attributes to token in a TokenStream (inside IncrementToken for example?).
>
> I'm working on entity extraction and want to flag specific tokens an
> entities, but I'm having problems.
>
> Thanks in advance,
> Itai
>
Here's a simple example of a filter that adds an atytribute saying 
whether a token is "the"

class YourTokenStream extends TokenFilter {
   private final YourAttribute att;
   private final CharTermAttribute term;
   private final TokenStream source;

   public YourTokenStream (TokenStream upstream) {
      att = addAttribute (YourAttribute.class);
      term = addAttribute (CharTermAttribute.class);
      source = upstream;
   }

   public boolean incrementToken () {
     if (source.incrementToken()) ?? {
       if ("the".equals (new String(term.buffer())) {
         att.setIsAnEnglishArticle(true);
         return true;
     }
     return false;
   }

}



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message