lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Hostetter <hossman_luc...@fucit.org>
Subject Re: Multiple Indices vs Single Index
Date Thu, 20 Sep 2007 22:32:38 GMT

: I was wondering if it will be better to just have 1 large index with all 
: the 40 indices combined.  I do not need to do dual-queries and my total 
: index size (if I create a single index) is about 3.4GB.  It will 
: increase to maximum of 5-6 GB.  I am running this on a dedicated machine 
: with 8GB RAM.

off the top of my head, there are 3 main reasons i can think of that would 
motivate one choice over another -- ultimately it's up to you...

1) FieldCache and sorting ... if all 40 sets of of Documents contain 
have consistently named fields, then there won't be much difference 
between 40 indexes and 1 index ... but if each of those 40 sets contain 
documents with radically differnet fields -- and you want to sort on N 
differnet fields for each sets -- then the total FieldCache sizes for each 
of those 40 indexes will be smaller then the FieldCaches for one gian 
index (because every document will get an entry wethe it makes sense or 
not.

2) idf statistics.  if you have common fields you search regardless of 
document set, the 40 index approach will maintain seperate sttistics -- 
this may be important if some terms are very common in only som docsets.  
the word "albino" may be really common in docset A but only one doc in 
docset B has it ... in the 40 index appraoch querying B for (albino 
elephant) will give a lot of weight to albino because it's so rare, but in 
the single index case albino may not be considered as significant because 
of ht unified idf value for all docsets 9even if hte query is constrained 
to docset B) ... again: this only matters if the fields overlap, if every 
docset has a unique set of fields then the idfs will be unique because 
they are by field)

3) management: it's probably a lot simpler to maintain and manage code 
that deals with one index then code that deals with 40 indexes. you milage 
may vary.


-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message