lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anand Kishore <andy.kish...@gmail.com>
Subject Indexing-Searching Design
Date Tue, 04 Oct 2005 17:05:37 GMT
Hi all,

Having read the mail in the mailing list archive about Best
Indexing-Searching Practices I have come up with the following architecture
for my application. Kindly evaluate and comment regarding the same.

Figure:

http://www.flickr.com/photos/28219682@N00/49301053/

Explanation:

The primary indexer (daemon) recieves the documents to be indexed. It
dispatches the documents to one of the secondary indexer nodes (via load
balancing). These indexing nodes index the documents in the RAMDirectory,
periodically writing it to a local index in the filesystem.

A cron process running on the central server (which contains the main index)
periodically checks for any new/updated indexes (small in size) on the
secondary nodes. It copies these new (small) indexes to the central server
(based on 'push changes onto main index'). An optimizer process running on
the central server periodically merges/optimizes the main index with the
smaller newer indexes. It also creates a checkpoint of the consistent index
everytime it performs optimization (the index.DATE approach).

The 'updaters' (cron processes on searcher nodes) periodically copy new
checkpoints via rsync onto their local system and create symbolic links to
them (same as proposed and used by Doug for Technorati).


--
- Andy

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message