lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Willnauer <simon.willna...@googlemail.com>
Subject Re: Re: Scale up design
Date Mon, 20 Dec 2010 08:41:12 GMT
On Mon, Dec 20, 2010 at 8:39 AM, Ganesh <emailgane@yahoo.co.in> wrote:
> I have done some benchmarking and based on that my estimate of RAM requirement would
be 3 - 4 GB. My question is to go for 64 bit or scale out with 3 systems?
What keeps you from moving to 64bit, I mean if you have those RAM req.
for JAVA HeapSpace I don't see much of a choice. You could try some
address space extensions which are around but I don't think its worth
it. Any reason why you hesitate?

simon
>
> Regards
> Ganesh
>
> ----- Original Message -----
> From: "Toke Eskildsen" <te@statsbiblioteket.dk>
> To: <java-user@lucene.apache.org>
> Sent: Thursday, December 16, 2010 4:20 PM
> Subject: Re: Re: Scale up design
>
>
>> On Thu, 2010-12-16 at 06:59 +0100, Ganesh wrote:
>>> 250 GB of data, 40 GB of Index Size, 60 million records is
>>> working fine with 1 GB RAM. We are storing minmal amount
>>> of data in index. We are doing sorting on Date. Even in
>>> single system, the database are shard.
>>
>> Looking back in the list, I see that you're sharding on weeks with 50+
>> weeks in the index.
>>
>>> build hosted solution. This stats will
>>> increase by minimum 10 times in 2 - 3 years. I plan to use
>>> 64 Bit, with 8 - 10 GB RAM allocated to JVM.
>>
>> When making a conservative estimate and multiplying with 10, you must
>> remember to do the same for the system memory available for disk cache.
>>
>> If your shards are searched sequentially, you could measure the response
>> time for a single shard (after warm up and with different queries), then
>> create a test-shard by merging 10 shards and measure response-time for
>> that. Subtracting the two numbers (to remove the overhead of the
>> front-end layer) and multiplying with 50 should give you a rough
>> estimate for the performance of an upscaled setup.
>>
>> Another measurement suggestion: Divide the current performance of the
>> full setup with the performance of a single shard, then multiply the
>> performance of a single created by merging 10 shards with that number.
>>
>> Regards,
>> Toke Eskildsen
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
> Send free SMS to your Friends on Mobile from your Yahoo! Messenger. Download Now! http://messenger.yahoo.com/download.php
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message