lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boogie Shafer <Boogie.Sha...@proquest.com>
Subject Re: how to debug solr performance degradation
Date Wed, 25 Feb 2015 20:01:42 GMT
rebecca,

you probably need to dig into your queries, but if you want to force/preload the index into
memory you could try doing something like

cat `find /path/to/solr/index` > /dev/null


if you haven't already reviewed the following, you might take a look here
https://wiki.apache.org/solr/SolrPerformanceProblems

perhaps going back to a very vanilla/default solr configuration and building back up from
that baseline to better isolate what might specific setting be impacting your environment

________________________________________
From: Tang, Rebecca <Rebecca.Tang@ucsf.edu>
Sent: Wednesday, February 25, 2015 11:44
To: solr-user@lucene.apache.org
Subject: RE: how to debug solr performance degradation

Sorry, I should have been more specific.

I was referring to the solr admin UI page. Today we started up an AWS
instance with 240 G of memory to see if we fit all of our index (183G) in
the memory and have enough for the JMV, could it improve the performance.

I attached the admin UI screen shot with the email.

The top bar is ³Physical Memory² and we have 240.24 GB, but only 4% 9.52
GB is used.

The next bar is Swap Space and it¹s at 0.00 MB.

The bottom bar is JVM Memory which is at 2.67 GB and the max is 26G.

My understanding is that when Solr starts up, it reserves some memory for
the JVM, and then it tries to use up as much of the remaining physical
memory as possible.  And I used to see the physical memory at anywhere
between 70% to 90+%.  Is this understanding correct?

And now, even with 240G of memory, our index is performing at 10 - 20
seconds for a query.  Granted that our queries have fq¹s and highlighting
and faceting, I think with a machine this powerful I should be able to get
the queries executed under 5 seconds.

This is what we send to Solr:
q=(phillip%20morris)
&wt=json
&start=0
&rows=50
&facet=true
&facet.mincount=0
&facet.pivot=industry,collection_facet
&facet.pivot=availability_facet,availabilitystatus_facet
&facet.field=dddate
&fq%3DNOT(pg%3A1%20AND%20(dt%3A%22blank%20document%22%20OR%20dt%3A%22blank%
20page%22%20OR%20dt%3A%22file%20folder%22%20OR%20dt%3A%22file%20folder%20be
gin%22%20OR%20dt%3A%22file%20folder%20cover%22%20OR%20dt%3A%22file%20folder
%20end%22%20OR%20dt%3A%22file%20folder%20label%22%20OR%20dt%3A%22file%20she
et%22%20OR%20dt%3A%22file%20sheet%20beginning%22%20OR%20dt%3A%22tab%20page%
22%20OR%20dt%3A%22tab%20sheet%22))
&facet.field=dt_facet
&facet.field=brd_facet
&facet.field=dg_facet
&hl=true
&hl.simple.pre=%3Ch1%3E
&hl.simple.post=%3C%2Fh1%3E
&hl.requireFieldMatch=false
&hl.preserveMulti=true
&hl.fl=ot,ti
&f.ot.hl.fragsize=300
&f.ot.hl.alternateField=ot
&f.ot.hl.maxAlternateFieldLength=300
&f.ti.hl.fragsize=300
&f.ti.hl.alternateField=ti
&f.ti.hl.maxAlternateFieldLength=300
&fq={!collapse%20field=signature}
&expand=true
&sort=score+desc,availability_facet+asc


My guess is that it¹s performing so badly because it¹s only using 4% of
the memory? And searches require disk access.


Rebecca
________________________________________
From: Shawn Heisey [apache@elyograg.org]
Sent: Tuesday, February 24, 2015 5:23 PM
To: solr-user@lucene.apache.org
Subject: Re: how to debug solr performance degradation

On 2/24/2015 5:45 PM, Tang, Rebecca wrote:
> We gave the machine 180G mem to see if it improves performance.  However,
> after we increased the memory, Solr started using only 5% of the physical
> memory.  It has always used 90-something%.
>
> What could be causing solr to not grab all the physical memory (grabbing
> so little of the physical memory)?

I would like to know what memory numbers in which program you are
looking at, and why you believe those numbers are a problem.

The JVM has a very different view of memory than the operating system.
Numbers in "top" mean different things than numbers on the dashboard of
the admin UI, or the numbers in jconsole.  If you're on Windows, then
replace "top" with task manager, process explorer, resource monitor, etc.

Please provide as many details as you can about the things you are
looking at.

Thanks,
Shawn


Mime
View raw message