lucene-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: cache managing with frequent update
Date Sat, 11 Apr 2009 00:30:45 GMT
At Veoh, we pulled machines from the pool during updates, but we did the
updates by copying entirely new indexes to the machines.  We might have been
able to avoid pulling the machines from the pool if we had used better I/O
scheduling, but we were very risk averse by that time.

My current system uses elastic computing instead based on Katta.  When doing
a full update, we just launch an entire new search cluster and then flip
traffic to it.  Nodes download shards autonomously from an HDFS file
system.  Updates in the form of additions (our dominant form) are propagated
by simply adding a new shard and letting Katta distribute it.  This seems to
work *much* better than the Solr based system we had a Veoh.

On Fri, Apr 10, 2009 at 3:18 AM, sunnyfr <johanna.34@gmail.com> wrote:

> How people do when they have frequent update to do ? to they turn off their
> servers during the warmup ???
> what is the solution, should I make it lower ??
>



-- 
Ted Dunning, CTO
DeepDyve

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message