lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Adding new documents to index
Date Wed, 19 Mar 2008 22:48:38 GMT
Hi,

Adding new documents to the index is not very costly even when the  
index is large.  The newly added documents result in additional  
segment(s).  If you then optimize then that is extremely costly for a  
large index.  Test your search performance to see if you really need  
to optimize.

The costly part is re-opening your searchers.  There are plans  
underway to improve the re-open performance so that the cost is more  
in proportion to what new segments were newly added (unlike today  
where cost is in proportion to total size of the index), but those  
are just plans at this point.

Mike

Vinicius Carvalho wrote:
> Hello there! Since I've just begun with lucene, some concepts are  
> kinda new
> for me :). One of the is the whole indexing process. Well, AFAIK,  
> indexing
> should happen in a batch process right, to maximize the time spent  
> on this
> operation. One issue tough, is that our client wants "instants search
> results", I mean, after something is added, the next search should  
> have it
> available. Well, it's almost impossible to convince him (and I  
> believe you
> guys know how this works) that indexing should not happen online  
> (maybe I'm
> just wrong here as well).
>
> Well, my question is, when I have a reasonable amount of indexed  
> data, how
> expensive is to add a new document? Does it re-builds the entire  
> index? I
> believe we'll have a small amount of data (when compare to things  
> I've seen
> lucene is capable of), probably 50k of html docs, 10k of pdf, and 5--6
> million rows of database information. With that scenario, an online  
> index
> (plus optimize) would be a good idea?
>
> Best regards
>
> -- 
> "In a world without fences and walls, who needs Gates and Windows?"


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message