lucene-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sunnyfr <johanna...@gmail.com>
Subject commit often and lot of data cost too much?
Date Tue, 31 Mar 2009 13:51:32 GMT

Hi,

I've about 14M of document. My index is about 11G.
For the moment I update every 20mn about 30 000 documents. 
Lucene alwarys merge data, What would you reckon?
My replication cost too much for the slave, they always bring back new index
directories and no segment.

Is there a way to get around this issue ? what would you reckon to people
who need fresh update on the slave with a big amount of data ?? 
Thanks a lot,


http://www.lucidimagination.com/Community/Hear-from-the-Experts/Articles/Scaling-Lucene-and-Solr/
:
"When the time and bandwidth needed for replication is less of a concern,
and high query throughput is more important, it can be wise to abandon the
advantage of transferring changed segments and only replicate fully
optimized indexes. It costs a bit more in terms of resources, but the master
will eat the cost of optimizing (so that users don't see the standard
machine slowdown affect that performing an optimize brings), and the slaves
will always get a fully optimized index to issue queries against, allowing
for maximum query performance. Generally, bandwidth for replication is not
much of a concern now, but keep in mind that optimizing on a large index can
be quite time consuming, so this strategy is not for every situation."
-- 
View this message in context: http://www.nabble.com/commit-often-and-lot-of-data-cost-too-much--tp22804941p22804941.html
Sent from the Lucene - General mailing list archive at Nabble.com.


Mime
View raw message