lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amit Nithian <anith...@gmail.com>
Subject Re: Hardware Specs Question
Date Tue, 31 Aug 2010 03:34:13 GMT
Lance,

makes sense and I have heard about the long GC times on large heaps but I
personally haven't experienced a slowdown but that doesn't mean anything
either :-). Agreed that tuning the SOLR caching is the way to go.

I haven't followed all the solr/lucene changes but from what I remember
there are synchronization points that could be a bottleneck where adding
more cores won't help this problem? Or am I completely missing something.

Thanks again
Amit

On Mon, Aug 30, 2010 at 8:28 PM, scott chu (朱炎詹) <scott.chu@udngroup.com>wrote:

> I am also curious as Amit does. Can you make an example about the garbage
> collection problem you mentioned?
>
> ----- Original Message ----- From: "Lance Norskog" <goksron@gmail.com>
> To: <solr-user@lucene.apache.org>
> Sent: Tuesday, August 31, 2010 9:14 AM
> Subject: Re: Hardware Specs Question
>
>
>
>  It generally works best to tune the Solr caches and allocate enough
>> RAM to run comfortably. Linux & Windows et. al. have their own cache
>> of disk blocks. They use very good algorithms for managing this cache.
>> Also, they do not make long garbage collection passes.
>>
>> On Mon, Aug 30, 2010 at 5:48 PM, Amit Nithian <anithian@gmail.com> wrote:
>>
>>> Lance,
>>>
>>> Thanks for your help. What do you mean by that the OS can keep the index
>>> in
>>> memory better than Solr? Do you mean that you should use another means to
>>> keep the index in memory (i.e. ramdisk)? Is there a generally accepted
>>> heap
>>> size/index size that you follow?
>>>
>>> Thanks
>>> Amit
>>>
>>> On Mon, Aug 30, 2010 at 5:00 PM, Lance Norskog <goksron@gmail.com>
>>> wrote:
>>>
>>>  The price-performance knee for small servers is 32G ram, 2-6 SATA
>>>> disks on a raid, 8/16 cores. You can buy these servers and half-fill
>>>> them, leaving room for expansion.
>>>>
>>>> I have not done benchmarks about the max # of processors that can be
>>>> kept busy during indexing or querying, and the total numbers: QPS,
>>>> response time averages & variability, etc.
>>>>
>>>> If your index file size is 8G, and your Java heap is 8G, you will do
>>>> long garbage collection cycles. The operating system is very good at
>>>> keeping your index in memory- better than Solr can.
>>>>
>>>> Lance
>>>>
>>>> On Mon, Aug 30, 2010 at 4:52 PM, Amit Nithian <anithian@gmail.com>
>>>> wrote:
>>>> > Hi all,
>>>> >
>>>> > I am curious to know get some opinions on at what point having more
>
>>>> CPU
>>>> > cores shows diminishing returns in terms of QPS. Our index size is >
>>>> about
>>>> 8GB
>>>> > and we have 16GB of RAM on a quad core 4 x 2.4 GHz AMD Opteron 2216.
>>>> > Currently I have the heap to 8GB.
>>>> >
>>>> > We are looking to get more servers to increase capacity and because
>
>>>> the
>>>> > warranty is set to expire on our old servers and so I was curious >
>>>> before
>>>> > asking for a certain spec what others run and at what point does >
>>>> having
>>>> more
>>>> > cores cease to matter? Mainly looking at somewhere between 4-12 cores
>>>> > per
>>>> > server.
>>>> >
>>>> > Thanks!
>>>> > Amit
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Lance Norskog
>>>> goksron@gmail.com
>>>>
>>>>
>>>
>>
>>
>> --
>> Lance Norskog
>> goksron@gmail.com
>>
>>
>
>
> --------------------------------------------------------------------------------
>
>
>
> ___b___J_T_________f_r_C
> Checked by AVG - www.avg.com
> Version: 9.0.851 / Virus Database: 271.1.1/3102 - Release Date: 08/30/10
> 14:35:00
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message