Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of rcmuir@gmail.com designates
 209.85.216.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=QmcY5wJIQpzElag4FS5H2nyPUg22zw9ox/Q/QGQMP4PYFaWIr0xKBcw9WoP8PIcq6W
         CPWBvND88qITCBJM9aZhv0JflwwU9KCIRTN45b0Ngl5msIyuqc1DHy7E5UO7ThA6cgad
         5/VlAhjfMZL8CXCE+OE1cOHWFvdjuBdn00Xjw=
MIME-Version: 1.0
In-Reply-To: <C01848B5-150D-4D16-A7E7-B67B9F0A20BB@opensourceconnections.com>
References: <C01848B5-150D-4D16-A7E7-B67B9F0A20BB@opensourceconnections.com>
From: Robert Muir <rcmuir@gmail.com>
Date: Tue, 24 Aug 2010 12:18:32 -0400
Message-ID: <AANLkTinQpa_sXgUqiY2RppGXW3nJhsL+gktzMD_bsyTZ@mail.gmail.com>
Subject: Re: Should analysis.jsp honor maxFieldLength
To: dev@lucene.apache.org
Content-Type: multipart/alternative; boundary=00c09fa00143eeb36d048e941e35

--00c09fa00143eeb36d048e941e35
Content-Type: text/plain; charset=UTF-8

On Tue, Aug 24, 2010 at 12:03 PM, Eric Pugh <epugh@opensourceconnections.com
> wrote:

> Hi all,
>
> I have maxFieldLength set to 10000 in solrconfig.xml, but was playing
> around with really large document (The King James Bible) in analysis.jsp.
> I hacked analysis.jsp to show me the number of terms at each filter, and the
> headers, but without turning everything on by checkboxing verbose.
>
> My results shown at this screenshot:
> http://img.skitch.com/20100824-t36rq45i2wfimwyd53gwiqebdy.png seem to
> confirm that maxFieldLength is NOT honored by the analysis.jsp.
>
>
Separate from whether or not analysis.jsp should do this (I happen to think
the closer to "reality" it is, the better), I think the easiest
implementation would be to wrap the entire stream with
LimitTokenCountFilter:

/**
 * This TokenFilter limits the number of tokens while indexing. It is
 * a replacement for the maximum field length setting inside {@link
org.apache.lucene.index.IndexWriter}.
 */

If i remember, its not exactly the same as the maxFieldLength, but its
pretty close.

-- 
Robert Muir
rcmuir@gmail.com

--00c09fa00143eeb36d048e941e35
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<br><div class=3D"gmail_quote">On Tue, Aug 24, 2010 at 12:03 PM, Eric Pugh =
<span dir=3D"ltr">&lt;<a href=3D"mailto:epugh@opensourceconnections.com">ep=
ugh@opensourceconnections.com</a>&gt;</span> wrote:<br><blockquote class=3D=
"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding=
-left:1ex;">

Hi all,<br>
<br>
I have maxFieldLength set to 10000 in solrconfig.xml, but was playing aroun=
d with really large document (The King James Bible) in analysis.jsp. =C2=A0=
 I hacked analysis.jsp to show me the number of terms at each filter, and t=
he headers, but without turning everything on by checkboxing verbose.<br>


<br>
My results shown at this screenshot: <a href=3D"http://img.skitch.com/20100=
824-t36rq45i2wfimwyd53gwiqebdy.png" target=3D"_blank">http://img.skitch.com=
/20100824-t36rq45i2wfimwyd53gwiqebdy.png</a> seem to confirm that maxFieldL=
ength is NOT honored by the analysis.jsp.<br>

<br></blockquote><div><br></div><div>Separate from whether or not analysis.=
jsp should do this (I happen to think the closer to &quot;reality&quot; it =
is, the better), I think the easiest implementation would be to wrap the en=
tire stream with LimitTokenCountFilter:</div>

<div><br></div><div>/**</div><div>=C2=A0* This TokenFilter limits the numbe=
r of tokens while indexing. It is</div><div>=C2=A0* a replacement for the m=
aximum field length setting inside {@link org.apache.lucene.index.IndexWrit=
er}.</div>

<div>=C2=A0*/</div><div>=C2=A0</div><div>If i remember, its not exactly the=
 same as the maxFieldLength, but its pretty close.</div><div><br></div></di=
v>-- <br>Robert Muir<br><a href=3D"mailto:rcmuir@gmail.com">rcmuir@gmail.co=
m</a><br>


--00c09fa00143eeb36d048e941e35--