Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of jason.rutherglen@gmail.com
 designates 209.85.221.171 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=uDzKPcDU4eBM/7JLktnDM8gs3etIanHTGnSxW6s6hFLv0oGUjZHMwdoXWQVpac+J/0
         4H/a9Sytm2aa3VBqdSdDqhcUyJRHTnESk4W7sAZgCJO88smxxzYOEL05QCKtcy1te0/P
         grG+y5FOoaglDWNTef2o6JRtLFlgYwrNv7uNc=
MIME-Version: 1.0
In-Reply-To: <21228090C6E946F9873B45192743E48D@VEGA>
References: <21228090C6E946F9873B45192743E48D@VEGA>
Date: Thu, 10 Sep 2009 15:27:49 -0700
Message-ID: <85d3c3b60909101527i7374c20cy8b629d88142976fe@mail.gmail.com>
Subject: Re: Problem with CharStream and Tokenizers with custom reset(Reader)
	method
From: Jason Rutherglen <jason.rutherglen@gmail.com>
To: java-dev@lucene.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I've been seeing strange behavior perhaps related to this? Where
sometimes a query is parsed and analyzed using Solr analyzers to
it's first clause fairly randomly, and other times the same
exact query is parsed and analyzed to the full correct query with all
clauses. It's so baffling I haven't really figured out an
approach to debugging it. I wonder if it's related to this
stream resetting issue.

On Thu, Sep 10, 2009 at 7:54 AM, Uwe Schindler <uwe@thetaphi.de> wrote:
> When reviewing the new CharStream code added to Tokenizers, I found a
> serious problem with backwards compatibility and other Tokenizers, that d=
o
> not override reset(CharStream).
>
> The problem is, that e.g. CharTokenizer only overrides reset(Reader):
>
> =A0public void reset(Reader input) throws IOException {
> =A0 =A0super.reset(input);
> =A0 =A0bufferIndex =3D 0;
> =A0 =A0offset =3D 0;
> =A0 =A0dataLen =3D 0;
> =A0}
>
> If you reset such a Tokenizer with another CharStream (not a Reader), thi=
s
> method will never be called and breaking the whole Tokenizer.
>
> As CharStream extends Reader, I propose to remove this reset(CharStream
> method) and simply do an instanceof check to detect if the supplied Reade=
r
> is no CharStream and wrap it. We could also remove the extra ctor (becaus=
e
> most Tokenizers have no support for passing CharStreams). If the ctor als=
o
> checks with instanceof and warps as needed the code is backwards compatib=
le
> and we do not need to add additional ctors in subclasses.
>
> As this instanceof check is always done in CharReader.get() why not remov=
e
> ctor(CharStream) and reset(CharStream) completely?
>
> Any thoughts?
>
> I would like to fix this somehow before RC4, I', sorry :(
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org