lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ganesh" <emailg...@yahoo.co.in>
Subject Re: Re: Scale up design
Date Wed, 22 Dec 2010 05:01:30 GMT
Hello Simon,

I don't hesitate to move to 64 bit. I require a suggestion whether to move to 64 bit (Scale
up) or scale out with multiple system. I have started investigating 64 bit,  i want to know
about its performance and if anyone in this group has already tried using it.

Regards
Ganesh

----- Original Message ----- 
From: "Simon Willnauer" <simon.willnauer@googlemail.com>
To: <java-user@lucene.apache.org>
Sent: Monday, December 20, 2010 2:11 PM
Subject: Re: Re: Scale up design


> On Mon, Dec 20, 2010 at 8:39 AM, Ganesh <emailgane@yahoo.co.in> wrote:
>> I have done some benchmarking and based on that my estimate of RAM requirement would
be 3 - 4 GB. My question is to go for 64 bit or scale out with 3 systems?
> What keeps you from moving to 64bit, I mean if you have those RAM req.
> for JAVA HeapSpace I don't see much of a choice. You could try some
> address space extensions which are around but I don't think its worth
> it. Any reason why you hesitate?
> 
> simon
>>
>> Regards
>> Ganesh
>>
>> ----- Original Message -----
>> From: "Toke Eskildsen" <te@statsbiblioteket.dk>
>> To: <java-user@lucene.apache.org>
>> Sent: Thursday, December 16, 2010 4:20 PM
>> Subject: Re: Re: Scale up design
>>
>>
>>> On Thu, 2010-12-16 at 06:59 +0100, Ganesh wrote:
>>>> 250 GB of data, 40 GB of Index Size, 60 million records is
>>>> working fine with 1 GB RAM. We are storing minmal amount
>>>> of data in index. We are doing sorting on Date. Even in
>>>> single system, the database are shard.
>>>
>>> Looking back in the list, I see that you're sharding on weeks with 50+
>>> weeks in the index.
>>>
>>>> build hosted solution. This stats will
>>>> increase by minimum 10 times in 2 - 3 years. I plan to use
>>>> 64 Bit, with 8 - 10 GB RAM allocated to JVM.
>>>
>>> When making a conservative estimate and multiplying with 10, you must
>>> remember to do the same for the system memory available for disk cache.
>>>
>>> If your shards are searched sequentially, you could measure the response
>>> time for a single shard (after warm up and with different queries), then
>>> create a test-shard by merging 10 shards and measure response-time for
>>> that. Subtracting the two numbers (to remove the overhead of the
>>> front-end layer) and multiplying with 50 should give you a rough
>>> estimate for the performance of an upscaled setup.
>>>
>>> Another measurement suggestion: Divide the current performance of the
>>> full setup with the performance of a single shard, then multiply the
>>> performance of a single created by merging 10 shards with that number.
>>>
>>> Regards,
>>> Toke Eskildsen
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>> Send free SMS to your Friends on Mobile from your Yahoo! Messenger. Download Now!
http://messenger.yahoo.com/download.php
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
Send free SMS to your Friends on Mobile from your Yahoo! Messenger. Download Now! http://messenger.yahoo.com/download.php

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message