Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Message-ID: <1386837093.4260.166.camel@te-prime>
Subject: Re: Solr hardware memory question
From: Toke Eskildsen <te@statsbiblioteket.dk>
Reply-To: te@statsbiblioteket.dk
To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>
Date: Thu, 12 Dec 2013 09:31:33 +0100
In-Reply-To: 
 <CAE4tqLOUK2qLkbmcMrWVMq-9O3X9Kp8FsviYgWrbzeU1bNYUww@mail.gmail.com>
References: <74C97E7DF5A7784D997217FF75D1216612DDE59B@w2k3-bspex1>
	 <1386763339.4260.95.camel@te-prime>
	 <CAE4tqLOUK2qLkbmcMrWVMq-9O3X9Kp8FsviYgWrbzeU1bNYUww@mail.gmail.com>
Organization: State and University Library, Denmark
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0

On Thu, 2013-12-12 at 02:46 +0100, Joel Bernstein wrote:
> Curious how many documents per shard you were planning?

350-500 million, optimized to a single segment as the data are not
changing.

> The number of documents per shard and field type will drive the amount
> of a RAM needed to sort and facet. 

Very true. It makes a lot of sense to separate RAM requirements for the
Lucene/Solr structures and OS-caching.

It seems that Gil is working on about the same project as we are, so I
will elaborate in this thread:

We would like to perform some sort of grouping on URL, so that the same
page harvested at different points in time, is only displayed once. This
is probably the heaviest functionality as the cardinality of the field
will be near the number of documents.

For plain(er) faceting, things like MIME-type, harvest date and site
seems relevant. Those field have lower cardinality and they are
single-valued so the memory requirements are something like 
#docs*log2(#unique_values) bits
With 500M documents and 1000 values, that is 600MB. With 20 shards, we
are looking at 12GB per simple facet field.

Regards,
Toke Eskildsen