lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Danil ŢORIN <torin...@gmail.com>
Subject Re: Re: Scale up design
Date Wed, 22 Dec 2010 07:14:04 GMT
There are no noticeable performance gains/loses when moving to 64 bit,
assuming is the exactly same hardware (just 64bit OS), same index and
reasonable amount of java heap
(keep in mind that if you had 2gb on 32 bit you'll need almost 3gb on
64 bit due to lager pointer representation)

But once your index grows...heap grows, GC pauses might be quite
unpredictable on such large heaps.
That's why most people for 10 fold growth will recommend scale out,
instead of scale up.

But it depends a lot on how your index is structured, what kind of
queries you run, how often you update your index, and so on.

YMMV...the only to find out what works for you is to try it.

On Wed, Dec 22, 2010 at 07:01, Ganesh <emailgane@yahoo.co.in> wrote:
> Hello Simon,
>
> I don't hesitate to move to 64 bit. I require a suggestion whether to move to 64 bit
(Scale up) or scale out with multiple system. I have started investigating 64 bit,  i want
to know about its performance and if anyone in this group has already tried using it.
>
> Regards
> Ganesh
>
> ----- Original Message -----
> From: "Simon Willnauer" <simon.willnauer@googlemail.com>
> To: <java-user@lucene.apache.org>
> Sent: Monday, December 20, 2010 2:11 PM
> Subject: Re: Re: Scale up design
>
>
>> On Mon, Dec 20, 2010 at 8:39 AM, Ganesh <emailgane@yahoo.co.in> wrote:
>>> I have done some benchmarking and based on that my estimate of RAM requirement
would be 3 - 4 GB. My question is to go for 64 bit or scale out with 3 systems?
>> What keeps you from moving to 64bit, I mean if you have those RAM req.
>> for JAVA HeapSpace I don't see much of a choice. You could try some
>> address space extensions which are around but I don't think its worth
>> it. Any reason why you hesitate?
>>
>> simon
>>>
>>> Regards
>>> Ganesh
>>>
>>> ----- Original Message -----
>>> From: "Toke Eskildsen" <te@statsbiblioteket.dk>
>>> To: <java-user@lucene.apache.org>
>>> Sent: Thursday, December 16, 2010 4:20 PM
>>> Subject: Re: Re: Scale up design
>>>
>>>
>>>> On Thu, 2010-12-16 at 06:59 +0100, Ganesh wrote:
>>>>> 250 GB of data, 40 GB of Index Size, 60 million records is
>>>>> working fine with 1 GB RAM. We are storing minmal amount
>>>>> of data in index. We are doing sorting on Date. Even in
>>>>> single system, the database are shard.
>>>>
>>>> Looking back in the list, I see that you're sharding on weeks with 50+
>>>> weeks in the index.
>>>>
>>>>> build hosted solution. This stats will
>>>>> increase by minimum 10 times in 2 - 3 years. I plan to use
>>>>> 64 Bit, with 8 - 10 GB RAM allocated to JVM.
>>>>
>>>> When making a conservative estimate and multiplying with 10, you must
>>>> remember to do the same for the system memory available for disk cache.
>>>>
>>>> If your shards are searched sequentially, you could measure the response
>>>> time for a single shard (after warm up and with different queries), then
>>>> create a test-shard by merging 10 shards and measure response-time for
>>>> that. Subtracting the two numbers (to remove the overhead of the
>>>> front-end layer) and multiplying with 50 should give you a rough
>>>> estimate for the performance of an upscaled setup.
>>>>
>>>> Another measurement suggestion: Divide the current performance of the
>>>> full setup with the performance of a single shard, then multiply the
>>>> performance of a single created by merging 10 shards with that number.
>>>>
>>>> Regards,
>>>> Toke Eskildsen
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>> Send free SMS to your Friends on Mobile from your Yahoo! Messenger. Download
Now! http://messenger.yahoo.com/download.php
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
> Send free SMS to your Friends on Mobile from your Yahoo! Messenger. Download Now! http://messenger.yahoo.com/download.php
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message