Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of rahul.solr@gmail.com designates
 209.85.218.214 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=PmNonvL+2AZDWr9TDFLTcfSp6oAQZFLL9i7P9A3lB1Ad7EFnZj5wcuUXDoCLby5vA8
         iGu9dAgPH123bP+VS0Z6y4g3EwT+2knHrIqs77B2dQOOnmtjtLcNIqGYn3r3/BoN+kG2
         OH1TwNXAHebQRgWTYsJQ66O5BFn6dIvf079sU=
MIME-Version: 1.0
In-Reply-To: <42eef5910909290642g7c59c4eej5348158eb39702ed@mail.gmail.com>
References: <42eef5910909271241o292a7534h7c18b77859ea6be2@mail.gmail.com>
	 <c68e39170909271250x1d4a056do9a35de07ca1c4dbb@mail.gmail.com>
	 <42eef5910909280451l64f65ba2of97e345a94e5635e@mail.gmail.com>
	 <c68e39170909280618j6196de80w1fc14fcf18f4201e@mail.gmail.com>
	 <42eef5910909290642g7c59c4eej5348158eb39702ed@mail.gmail.com>
Date: Tue, 29 Sep 2009 19:28:18 +0530
Message-ID: <42eef5910909290658v316d85a5m3f668e569b8c39ac@mail.gmail.com>
Subject: Re: Measuring timing with debugQuery=true
From: Rahul R <rahul.solr@gmail.com>
To: solr-user@lucene.apache.org
Cc: yonik@lucidimagination.com
Content-Type: multipart/alternative; boundary=001485f7728c8820a00474b7ceaf

--001485f7728c8820a00474b7ceaf
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I just want to clarify here that I understand my memory allocation might be
less given the load on the system. The response times were only slightly
better when we ran the test on a Solaris box with 12CPU, 24G RAM and with
3.2 GB allocated for the JVM..... I know that I have a performance
problem. My main concern is to identify the reasons for the inconsistency
between the timing information shown between the debugQuery output (2.4s)
and the entire time taken by the EmbeddedSolrServer.query(SolrQuery)
function (16s). I feel that if I can find out where the remaining 13.6s get=
s
used, then I can look to improve accordingly. Thank you.

Regards
Rahul

On Tue, Sep 29, 2009 at 7:12 PM, Rahul R <rahul.solr@gmail.com> wrote:

> Sorry for the delayed response
>  **
> *How big are your documents?*
> I have totally 1 million documents. I have totally 1950 fields in the
> index. Every document would probably have values for around 20 - 50 field=
s.
>  *What is the total size of the index?*
> 1 GB
>
> *What's the amout of RAM on your box? How big is the JVM heap (and how
> much free memory is left on your system)?*
> I have 4 GB RAM. I am using Weblogic 10, 32 Bit. Since it is a windows bo=
x,
> I am able to allocate only 1 GB to the JVM. No other applications are
> running on the system. So the entire 4GB is at the disposal of the
> application. I am simulating load using a load tool (15 users)
>
> *Can you show what this slow query looks like (the whole request)?*
>
> q=3D*%3A*&rows=3D0&facet=3Dtrue&facet.mincount=3D1&facet.limit=3D2&f.S994=
2.facet.limit=3D100&facet.field=3DS9942&facet.field=3DS6878&facet.field=3DS=
9156&facet.field=3DS0369&facet.field=3DS9926&facet.field=3DS1421&facet.fiel=
d=3DS8990&facet.field=3DS6881&facet.field=3DS3552&debugQuery=3Dtrue
>
>
> q=3D*%3A*&fq=3DS9942%3A%22TEXAS+INSTRUMENTS%22&rows=3D0&facet=3Dtrue&face=
t.mincount=3D1&facet.limit=3D2&facet.field=3DS9942&facet.field=3DS6878&face=
t.field=3DS9156&facet.field=3DS0369&facet.field=3DS9926&facet.field=3DS1421=
&facet.field=3DS8990&facet.field=3DS6881&facet.field=3DS3552&debugQuery=3Dt=
rue
>
> Other information
> Solr 1.3, JDK 1.5.0_14
>
> regards
> Rahul
>
>   On Mon, Sep 28, 2009 at 6:48 PM, Yonik Seeley <
> yonik@lucidimagination.com> wrote:
>
>> On Mon, Sep 28, 2009 at 7:51 AM, Rahul R <rahul.solr@gmail.com> wrote:
>> > Yonik,
>> > I understand that the network can be a bottle-neck but I am pretty sur=
e
>> that
>> > it is not. I am operating on a 100 MBPS intranet....... How do I ensur=
e
>> that
>> > stored fields are cached by the OS ? Only the Solr caches within the J=
VM
>> are
>> > under my control...... The result set has around 10K documents of whic=
h
>> I am
>> > retrieving only 10......I am displaying a max of only 3 fields per
>> document
>> > in my result set. Can the reading time for these stored fields be so
>> long ?
>>
>> It could be a seek per document if the index is too big to fit in the
>> OS cache - but that still wouldn't be as slow as you report.
>> Something is fishy here.
>>
>> How big are your documents?
>> What is the total size of the index?
>> What's the amout of RAM on your box?
>> How big is the JVM heap (and how much free memory is left on your system=
)?
>> Can you show what this slow query looks like (the whole request)?
>>
>> > I have totally around 1 million documents in my index........ Any
>> thoughts
>> > on why the FacetComponent does not take any time while the
>> QueryComponent
>> > takes around 2.4s.
>>
>> It could be a field that has very few unique values and faceting just
>> completes quickly.
>> Make sure you're actually getting faceting data back (that it's
>> correctly turned on).
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>> > I am doing a faceted and keyword query ie I have both 'q'
>> > and 'fq' params in my query.... Thank you for your response.
>> >
>> > Regards
>> > Rahul
>> >
>> > On Mon, Sep 28, 2009 at 1:20 AM, Yonik Seeley <
>> yonik@lucidimagination.com>
>> > wrote:
>> >>
>> >> The response times in a Solr request don't include the time to read
>> >> stored fields (since the response is streamed) and doesn't include th=
e
>> >> time to transfer/read the response (which can be increased by a
>> >> slow/congested network link, or a slow client that doesn't read the
>> >> response immediately).
>> >>
>> >> How many documents are you retrieving?  Reading stored fields for
>> >> documents can be slow if they aren't cached by the OS since it's ofte=
n
>> >> a disk seek per document read for a large index.
>> >>
>> >> -Yonik
>> >> http://www.lucidimagination.com
>> >>
>> >>
>> >>
>> >> On Sun, Sep 27, 2009 at 3:41 PM, Rahul R <rahul.solr@gmail.com> wrote=
:
>> >> > Hello,
>> >> > I am trying to measure why some of my queries take a long time. I a=
m
>> >> > using
>> >> > EmbeddedSolrServer and with logging statements before and
>> >> > after the EmbeddedSolrServer.query(SolrQuery) function, I have foun=
d
>> the
>> >> > time to be around 16s. I added the debugQuery=3Dtrue and the timing
>> >> > component
>> >> > for this reads as following:
>> >> >
>> >> > *
>> >> >
>> >> >
>> timing:{time=3D2438.0,prepare=3D{time=3D0.0,org.apache.solr.handler.comp=
onent.QueryComponent=3D{time=3D0.0},org.apache.solr.handler.component.Facet=
Component=3D{time=3D0.0},org.apache.solr.handler.component.MoreLikeThisComp=
onent=3D{time=3D0.0},org.apache.solr.handler.component.HighlightComponent=
=3D{time=3D0.0},org.apache.solr.handler.component.DebugComponent=3D{time=3D=
0.0}},process=3D{time=3D2438.0,org.apache.solr.handler.component.QueryCompo=
nent=3D{time=3D2438.0},org.apache.solr.handler.component.FacetComponent=3D{=
time=3D0.0},org.apache.solr.handler.component.MoreLikeThisComponent=3D{time=
=3D0.0},org.apache.solr.handler.component.HighlightComponent=3D{time=3D0.0}=
,org.apache.solr.handler.component.DebugComponent=3D{time=3D0.0}}}
>> >> > *
>> >> >
>> >> > As you can see, this shows only 2.4s being used by the query. I can=
't
>> >> > seem
>> >> > to figure out where the rest of the time is being spent. This is
>> within
>> >> > my
>> >> > office intranet and I don't think the request-response time over th=
e
>> >> > wire
>> >> > will cause significant overhead. So my question : is the timing
>> >> > information
>> >> > presented here comprehensive or are there more time consuming
>> operations
>> >> > that are not represented here ? I guess GC pause times could be one
>> >> > answer
>> >> > (I hope not !).... Also, the above result was for a faceted query. =
I
>> >> > can't
>> >> > understand why the FacetComponent would be zero. Any thoughts ?
>> >> >
>> >> > Rahul
>> >> >
>> >
>> >
>>
>
>

--001485f7728c8820a00474b7ceaf--