lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Toke Eskildsen ...@statsbiblioteket.dk>
Subject Re: Re: Scale up design
Date Thu, 16 Dec 2010 10:50:38 GMT
On Thu, 2010-12-16 at 06:59 +0100, Ganesh wrote:
> 250 GB of data, 40 GB of Index Size, 60 million records is
> working fine with 1 GB RAM. We are storing minmal amount
> of data in index. We are doing sorting on Date. Even in
> single system, the database are shard.

Looking back in the list, I see that you're sharding on weeks with 50+
weeks in the index.

> build hosted solution. This stats will
> increase by minimum 10 times in 2 - 3 years. I plan to use
> 64 Bit, with 8 - 10 GB RAM allocated to JVM.

When making a conservative estimate and multiplying with 10, you must
remember to do the same for the system memory available for disk cache.

If your shards are searched sequentially, you could measure the response
time for a single shard (after warm up and with different queries), then
create a test-shard by merging 10 shards and measure response-time for
that. Subtracting the two numbers (to remove the overhead of the
front-end layer) and multiplying with 50 should give you a rough
estimate for the performance of an upscaled setup.

Another measurement suggestion: Divide the current performance of the
full setup with the performance of a single shard, then multiply the
performance of a single created by merging 10 shards with that number.

Regards,
Toke Eskildsen


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message