lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dennis Gearon <gear...@sbcglobal.net>
Subject Re: Improving Solr performance
Date Mon, 10 Jan 2011 01:38:42 GMT
These are definitely server grade machines.

There aren't any desktops I know of (that aren't made for HD video 
editing/rendition) that ever need that kind of memory.

 Dennis Gearon


Signature Warning
----------------
It is always a good idea to learn from your own mistakes. It is usually a better 
idea to learn from others’ mistakes, so you do not have to make them yourself. 
from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'


EARTH has a Right To Life,
otherwise we all die.



----- Original Message ----
From: Shawn Heisey <solr@elyograg.org>
To: solr-user@lucene.apache.org
Sent: Sun, January 9, 2011 4:34:08 PM
Subject: Re: Improving Solr performance

On 1/7/2011 2:57 AM, supersoft wrote:
> have deployed a 5-sharded infrastructure where: shard1 has 3124422 docs
> shard2 has 920414 docs shard3 has 602772 docs shard4 has 2083492 docs shard5
> has 11915639 docs Indexes total size: 100GB
> 
> The OS is Linux x86_64 (Fedora release 8) with vMem equal to 7872420 and I
> run the server using Jetty (from Solr example download) with: java -Xmx3024M
> -Dsolr.solr.home=multicore -jar start.jar
> 
> The response time for a query is around 2-3 seconds. Nevertheless, if I
> execute several queries at the same time the performance goes down
> inmediately: 1 simultaneous query: 2516ms 2 simultaneous queries: 4250,4469
> ms 3 simultaneous queries: 5781, 6219, 6219 ms 4 simultaneous queries: 6484,
> 7203, 7719, 7781 ms...

I see from your other messages that these indexes all live on the same machine.  
You're almost certainly I/O bound, because you don't have enough memory for the 
OS to cache your index files.  With 100GB of total index size, you'll get best 
results with between 64GB and 128GB of total RAM.  Alternatively, you could use 
SSD to store the indexes instead of spinning hard drives, or put each shard on 
its own physical machine with RAM appropriately sized for the index.  For shard5 
on its own machine, at 64GB index size, you might be able to get away with 32GB, 
but ideally you'd want 48-64GB.

Can you do anything to reduce the index size?  Perhaps you are storing fields 
that you don't need to be returned in the search results.  Ideally, you should 
only include enough information to fully populate a search results grid, and 
retrieve detail information for an individual document from the original data 
source instead of Solr.

Thanks,
Shawn

Mime
View raw message