lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nader Henein <>
Subject Re: Index Replication / Clustering
Date Mon, 27 Jun 2005 09:14:51 GMT
I implemented a JMS based solution about a year ago because I thought it 
would solve my atomicity problem and give me a centralized way of 
indexing, you'll have to use the pluggable persistence (if you use 
ActiveMQ) to be able to recover from a failure and you'll also need some 
way of maintaining which records have and haven't been indexed in a 
persistent store so you can run sanitization later on, assuming that 
every time you add a document to one of you clustered indecies 
everything goes well is a sure way of guaranteeing that you end up with 
n indecies with no two alike especially if you have a high volume 
indecies with high rates of change.

Nader Henein

Stephane Bailliez wrote:

> Hi Paul,
> Thanks for the reply. Many interesting points.
> Paul Smith wrote:
>> Why not try using JMS messaging to send messages to the indexing  
>> server that Document X needs to be updated via a JMS queue?  This  
>> gives you the flexibility to have the indexing system down but not  
>> lose the message that it needs to be indexed, and also allows the  
>> indexing server to be 'busy' without affecting the application that  
>> is performing the updates from slowing down too.
> Excellent idea.
>> If you use ActiveMQ for JMS, you can take advantage of it's 
>> Composite  Destination feature and have a virtual Queue/Topic that is 
>> actually  several Queues/Topics.  This is what we use to keep a 
>> mirror index  server completely in sync.  The application sends an 
>> update message  to a queue named "queue://index1, queue://index2", 
>> which becomes 2  separate queues for the 2 servers, allowing them to 
>> process the same  message whenever they can get around to it.
> Ah, the composite topic, is indeed a good nice. But out of 
> curiosity...did you put your 2 nodes (consumers) as embedded brokers 
> or is the producer as the main broker ?
>> We then place Apache in front of these 2 mirrored Index/Search nodes  
>> so the application can use web-services to query the search node  
>> without actually being aware that there is 2 of them behind the  
>> scenes, leaving Apache to do the load-balancing and fail-over as the  
>> index/search nodes come up/down without the main application knowing  
>> anything about it.
> Ideally, the 2 nodes have the same state when running.
> What happens when a node fails and that you put it back online and 
> that it needs to catch up with all missing messages in its queue ?
> Is it considered 'offline' until it catches up ? If yes how do you do 
> it ? If no, I guess you don't mind that a search request may not give 
> the same result depending on the node it is load-balanced, correct ?
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:
> ---


Nader S. Henein
Senior Applications Architect


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message