Mailing-List: contact lucene-dev-help@jakarta.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Lucene Developers List" <lucene-dev@jakarta.apache.org>
Message-Id: <s08508fb.093@gwia201.syr.edu>
Date: Tue, 20 Apr 2004 11:26:38 -0400
From: "Grant Ingersoll" <gsingers@syr.edu>
To: <lucene-dev@jakarta.apache.org>
Subject: Re: incorrect OO in lucene source?
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

The thread safety issues are on the search side usage of Analyzer, not =
indexing.

>>> tdv@part.net 04/20/04 10:52AM >>>
Grant Ingersoll wrote:

>I agree with Robert, as I have had similar wishes about more interface =
capabilities, but also agree with Eric in that Lucene works great in a lot =
of ways.    I have found the current design causes you to have to hard =
code things that shouldn't need to be hard coded, especially in the =
TokenStream area.  The idea of writing a new Analyzer every time you want =
to change a Tokenizer or TokenFilter is very limiting.  In my application =
I need the flexibility to re-index and evaluate fairly often.  The current =
Analyzer implementation would require me to write a new Analyzer for every =
experiment and that is not manageable.  Do others have this issue?
>
> =20
>
I had this issue. I have solved this by rewritting the API around=20
TokenStream (mainly introducing an interface that allows resetting the=20
source stream) and creating a generalized analyzer class. This analyzer=20
class holds a reference to the TokenStream pipeline to which it=20
delegates. A PerField analyzer is populated with Analyzers configured=20
from JNDI (essentially Tokenizer and TokenStreamDecorator compositions).=20=

When TokenStream(String fieldName, Reader reader) is called the analyzer=20=

resets its TokenStream reference before returning it.

>I submitted a "broken" patch that converts the analyzers and token =
streams to interfaces, but as Doug pointed out, it is not currently thread =
safe (I have another version that uses reflection that is thread safe).  I =
intend to go back and make it thread-safe, but haven't had the time.  =
Anyway, this patch contains an interface implementation of Analyzer and =
TokenStream that we may find useful in the future and if someone else =
wants to take up the ball and make it thread-safe, I don't think it would =
take too long.
> =20
>
What were the issues related to thread safety? Are invocations of an=20
analyzer within an IndexWriter not single threaded? I was unsure of=20
this, but planned to object pool my TokenStream compositions if needed.

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org=20
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org=20


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org