lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Toke Eskildsen>
Subject Re: Solr hardware memory question
Date Thu, 12 Dec 2013 08:31:33 GMT
On Thu, 2013-12-12 at 02:46 +0100, Joel Bernstein wrote:
> Curious how many documents per shard you were planning?

350-500 million, optimized to a single segment as the data are not

> The number of documents per shard and field type will drive the amount
> of a RAM needed to sort and facet. 

Very true. It makes a lot of sense to separate RAM requirements for the
Lucene/Solr structures and OS-caching.

It seems that Gil is working on about the same project as we are, so I
will elaborate in this thread:

We would like to perform some sort of grouping on URL, so that the same
page harvested at different points in time, is only displayed once. This
is probably the heaviest functionality as the cardinality of the field
will be near the number of documents.

For plain(er) faceting, things like MIME-type, harvest date and site
seems relevant. Those field have lower cardinality and they are
single-valued so the memory requirements are something like 
#docs*log2(#unique_values) bits
With 500M documents and 1000 values, that is 600MB. With 20 shards, we
are looking at 12GB per simple facet field.

Toke Eskildsen

View raw message