Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
From: Edoardo Causarano <edoardo.causarano@gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\))
Subject: Re: A question over TokenFilters
Date: Fri, 21 Apr 2017 17:55:10 +0200
References: <326B6D52-2008-4CE2-982E-2CF4953483F0@gmail.com>
 <52124279.1413064.1492788731521@mail.yahoo.com>
To: java-user@lucene.apache.org
In-Reply-To: <52124279.1413064.1492788731521@mail.yahoo.com>
Message-Id: <CBEBB836-5495-4A01-ADA4-8D5392C63F64@gmail.com>
archived-at: Fri, 21 Apr 2017 15:55:08 -0000

Hi,

thanks for your reply. In several other implementations I=E2=80=99ve =
seen this pattern of using a while(input.incrementToken()) within the =
filter=E2=80=99s incrementToken method. Is this approach recommended or =
are there hidden traps (eg: memory consumption, dependency on filter =
ordering and so on)=20


Best,
Edoardo

> On 21 Apr 2017, at 17:32, Ahmet Arslan <iorixxx@yahoo.com.INVALID> =
wrote:
>=20
> Hi,
> LimitTokenCountFilter is used to index first n tokens. May be it can =
inspire you.
>=20
> Ahmet
> On Friday, April 21, 2017, 6:20:11 PM GMT+3, Edoardo Causarano =
<edoardo.causarano@gmail.com> wrote:
> Hi all.
>=20
> I=E2=80=99m relatively new to Lucene, so I have a couple questions =
about writing custom filters.
>=20
> The way I understand it, one would extend =
org.apache.lucene.analysis.TokenFilter and override #incrementToken to =
examine the current token provided by a stream token producer.
>=20
> I=E2=80=99d like to write some logic that considers the last n seen =
tokens therefore I need to access this context as the filter chain is =
scanning the stream.
>=20
> Can anyone point to an example of such a construct?=20
>=20
> Also, how would I access and update this context keeping =
multithreading in mind? Actually, what is the treading model of a =
TokenStream, can anyone point out a good summary for it?
>=20
> TIA
>=20
>=20
> Best,
> Edoardo
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org