Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of sarowe@gmail.com designates
 209.85.212.51 as permitted sender)
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\))
Subject: Re: TokenStream: How to get token text?
From: Steve Rowe <sarowe@gmail.com>
In-Reply-To: 
 <CAMXF=YDPAwW-XC3V82GwxVGbG3_X6MZ2oK1LiQU3dwwwtC4b8w@mail.gmail.com>
Date: Tue, 25 Dec 2012 13:48:41 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <94BE591F-BB4D-40F6-86DE-3448A346AB08@gmail.com>
References: 
 <CAMXF=YDPAwW-XC3V82GwxVGbG3_X6MZ2oK1LiQU3dwwwtC4b8w@mail.gmail.com>
To: java-user@lucene.apache.org

Hi Dima,

Did you see my response to your earlier email?  I think it's what you're =
looking for:

http://markmail.org/message/jdcjxauj4odyuv7e

Steve

On Dec 25, 2012, at 1:17 PM, dokondr <dokondr@gmail.com> wrote:

> Hello,
> Please, help. I am lost in TokenStream / Token / Analyzer API.
> I am trying to figure out how to get _token_itself_ or token text =
while
> looking at "Invoking the Analyzer" example (see example below and also =
at:
> =
http://lucene.apache.org/core/4_0_0/core/org/apache/lucene/analysis/packag=
e-summary.html?is-external=3Dtrue#package_description
> )
>=20
> Method "ts.reflectAsString(true))" returns lots of useful info:
> =
org.apache.lucene.analysis.tokenattributes.CharTermAttribute#term=3Dsome,o=
rg.apache.lucene.analysis.tokenattributes.TermToBytesRefAttribute#bytes=3D=
[73
> 6f 6d
> =
65],org.apache.lucene.analysis.tokenattributes.OffsetAttribute#startOffset=
=3D0,org.apache.lucene.analysis.tokenattributes.OffsetAttribute#endOffset=3D=
4,org.apache.lucene.analysis.tokenattributes.PositionIncrementAttribute#po=
sitionIncrement=3D1,org.apache.lucene.analysis.tokenattributes.TypeAttribu=
te#type=3D<ALPHANUM>,org.apache.lucene.analysis.tokenattributes.KeywordAtt=
ribute#keyword=3Dfalse
>=20
> Yet, how to get token itself? In this case "some" ?
>=20
> Thanks!
>=20
> ------ Example in the documentation --------
>=20
>   Version matchVersion =3D Version.LUCENE_XY; // Substitute desired =
Lucene
> version for XY
>    Analyzer analyzer =3D new StandardAnalyzer(matchVersion); // or any =
other
> analyzer
>    TokenStream ts =3D analyzer.tokenStream("myfield", new =
StringReader("some
> text goes here"));
>    OffsetAttribute offsetAtt =3D addAttribute(OffsetAttribute.class);
>=20
>    try {
>      ts.reset(); // Resets this stream to the beginning. (Required)
>      while (ts.incrementToken()) {
>        // Use AttributeSource.reflectAsString(boolean)
>        // for token stream debugging.
>        System.out.println("token: " + ts.reflectAsString(true));
>=20
>        System.out.println("token start offset: " +
> offsetAtt.startOffset());
>        System.out.println("  token end offset: " + =
offsetAtt.endOffset());
>      }
>      ts.end();   // Perform end-of-stream operations, e.g. set the =
final
> offset.
>    } finally {
>      ts.close(); // Release resources associated with this stream.
>    }


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org