Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of a_rexha@hotmail.com designates
 157.55.1.175 as permitted sender)
Message-ID: <DUB123-W10E612D35D6021BAB2B34F97600@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_1a3ba8c0-c146-4dc8-a60a-16f322372483_"
From: andi rexha <a_rexha@hotmail.com>
To: "java-user@lucene.apache.org" <java-user@lucene.apache.org>
Subject: Getting new token stream from analyzer for legacy projects!
Date: Fri, 12 Dec 2014 16:29:33 +0100
Importance: Normal
MIME-Version: 1.0

--_1a3ba8c0-c146-4dc8-a60a-16f322372483_
Content-Type: text/plain; charset="iso-8859-2"
Content-Transfer-Encoding: quoted-printable

Hi=2C=20
I have a legacy problem with the token stream. In my application I create a=
 batch of documents from a unique analyzer (this due to configuration). I a=
dd the field using the tokenStream from the analyzer(for internal reasons).=
 In a pseudo code this translates in :=20

Analyzer analyzer =3D getFromConfig()=3B
Collection docsToIndex=3B=20


for (int i =3D 0=3B i < batchSize()=3B i ++) {
      Documet doc =3D new Document()=3B
      doc.add(field=2C analyzer.tokenStream("fieldName"=2C currentReader))=
=3B
      docsToIndex.add(doc)=3B
}

for (Document d : docToIndex) {
    indexWriter.add(doc)=3B
}


I get always an exception :
TokenStream contract violation: reset()/close() call missing=2C reset() cal=
led multiple times....

I understand that the analyzer creates one TokenStream per thread and that =
the TokenStream is used from the DefaultIndexingChain during the add docume=
nts=2C so the TokenStream is shared.=20

Is there a clean way I can overcome this problem? One possible way of cours=
e would be to get the token stream from a separate thread=2C but that would=
 be a dirty solution.=20


 		 	   		  =

--_1a3ba8c0-c146-4dc8-a60a-16f322372483_--