lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jake Mannix <>
Subject Re: How to setup a scalable deployment?
Date Wed, 07 Oct 2009 02:07:26 GMT
Hi Chris,

  Answering your question depends in part whether your kind of scalability
is dependent on sharding (your index size is expected to grow to very large)
or just replication (your query load is large, and you need failover).  It
sounds like you're mostly thinking about the latter.

1) Each web server indexes the content separately. This will potentially
> cause different web servers to have slightly different indexes at any given
> time and also duplicates the work load of indexing the content

  If your indexing throughput is small enough, this can be a perfectly
way to do this.  Just make sure you're not DOS'ing your DB if you're
via direct DB queries (ie. if you have a message queue or something else
firing off indexing events, instead of all web servers firing off
identical DB queries from different places.  DB caching will deal with this
pretty well, but you still need to be careful).

> 2) Using rsync (or a similar tool) to regularly update a local lucene index
> directory on each web server. I imagine there will be locking issues that
> need to be resolved here.

  rsync can work great, and and as Jason said, it is how Solr works and it
scales great.  Locking isn't really a worry, because in this setup, the
are read-only in this case, so they won't compete with rsync for write

> 3) Using a network file system that all the web servers can access. I don't
> have much experience in this area, but potentially latency on searches will
> be high?

  Generally this is a really bad idea, and can lead to really hard-to-debug
performance problems.

> 4) Some alternative lucene specific solution that I haven't found in the
> wiki / lucene documentation.

> The indexes aim to be as real-time as possible, I currently update my
> IndexReaders in a background thread every 20 seconds.

This is where things diverge from common practice, especially if you at some
point decide to lower that to 10 or 5 seconds.

In this case, I'd say that if you have a reliable, scalable queueing system
getting indexing events distributed to all of your servers, then indexing on
all replicas simultaneously can be the best way to have maximally realtime
search, either using the very new feature of "near realtime search" in
Lucene 2.9, by using something home-brewed which indexes in memory
and on disk simultaneously, or using Zoie ( ),
an open-source realtime search library built on top of Lucene which we at
LinkedIn built and have been using in production for serving tens of
millions of queries a day in real time (meaning milliseconds, even under
fairly high indexing load) for the past year.

  -jake mannix

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message