lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colin R <colin.russ...@dasmail.co.uk>
Subject Newbie Question: Master Index or 100s Small Index
Date Wed, 19 Mar 2014 10:55:46 GMT
We run a central database of 14M (and growing) photos with dates, captions,
keywords, etc. 

We currently upgrading from old Lucene Servers to latest Solr running with a
couple of dedicated  servers (6 core, 36GB, 500SSD). Planning on using Solr
Cloud.

We take in thousands of changes each day (big and small) so indexing may be
a bigger problem than searching.

My question is an architecture one.

These photos are currently indexed and searched in three ways.

1: The 14M pictures from above are split into a few hundred indexes that
feed a single website. This means index sizes of between 100 and 500,000
entries each.

2: 95% of these same photos are also wanted for searching on a global site.
Index size of 12M plus.

3: 80% of these same photos are also required for smaller group sites. Index
sizes of between 400K and 4M.

We currently make changes the single indexes and then merge into groups and
global. Due to the size of the numbers, is it worth changing or not.

Is it quicker/better to just have one big 14M index and filter the
complexities for each website or is it better to still maintain hundreds of
indexes so we are searching smaller one. Bear in mind, we get thousands of
changes a day PLUS very busy search servers.

Thanks

Col



--
View this message in context: http://lucene.472066.n3.nabble.com/Newbie-Question-Master-Index-or-100s-Small-Index-tp4125407.html
Sent from the Solr - User mailing list archive at Nabble.com.

Mime
View raw message