lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <>
Subject RE: Stop words filter
Date Wed, 23 Jun 2010 06:35:47 GMT
Hi Vinicius,

You should read the Package-Level Docs:

To get the Token attributes, you have to add Attributes to your TokenStream
using addAttribute() and then you have easy access to the various attributes
of each token, when iterating with incrementToken().

If you want to program an own Tokenizer, start by inspecting a provided one
and do it similar. Also test cases for existing analyzers are a good way to
look into the usage. A good method to test TokenStream/Analyzers are in the
test-package's class BaseTokenStreamTestCase: assertTokenStreamContents(),

Also the Lucene In Action *2* book gives good examples.


Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen

> -----Original Message-----
> From: Vinicius Carvalho []
> Sent: Wednesday, June 23, 2010 4:50 AM
> To:
> Subject: Stop words filter
> Hello there! I've been using lucene as a Fult Text Search solution for
> time. And  although I'm familiar with Analyzers and Stemmers I never used
> them directly.
> I'm testing a few experiments on Sentiment Analysis and our
> implementation needs to perform stemming and stop word removal. I
> thought using lucene built-in support to spare me some coding time.
> Is there any example? I'm trying
> TokenStream stream = analyzer.tokenStream("", new
> StringReader(inputStr));
> Problem is that I could not find a way to get the result tokens. I was
> expecting something like stream.getTokens:Token[] :P
> Could someone point me in the right direction?
> Regards
> --
> The intuitive mind is a sacred gift and the rational mind is a faithful
> We have created a society that honors the servant and has forgotten the

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message