lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lee Turner" <Lee.Tur...@oyster.com>
Subject RE: Strategies for updating indexes.
Date Tue, 05 Apr 2005 08:53:53 GMT
Hi

Thank you for replying so quickly.  I am very pleased as I have just started down the road
of implementing a solution which is very nearly exactly like the one you describe below. 
It is good to know that I am not heading down a dead end.  I hadn't thought about the re-indexing
thread pausing the queue to stop it processing while the re-indexing takes place.  I will
also take a look at quartz.

Your input is very much appreciated

Many thanks
Lee


-----Original Message-----
From: Jens Kraemer [mailto:kraemer@webit.de] 
Sent: 05 April 2005 09:30
To: java-user@lucene.apache.org
Subject: Re: Strategies for updating indexes.

Hi,
please see comments below.

On Tue, Apr 05, 2005 at 08:38:04AM +0100, Lee Turner wrote:
> Hi
> 
> I was wondering whether anyone has any experience of multithreaded
> updates to indexes.  I the web app I am working on there are additions,
> updates and deletes that need to happen to the index throughout the
> runtime of the application.  Also, the application is run in a cluster
> with each app server having its own index.  This means that periodically
> each app server is going to have to go through a re-indexing process to
> make sure that its index has all the changes from the other app servers
> in it.  This process can take a few seconds so if another update to the
> index occurs at this time it will need to be queued in some way to make
> sure it happens after the re-indexing.
> 
> I was just wondering if anyone had any pointers for doing this kind of
> thing.  Any help would be gratefully appreciated.

I usually have a service class wrapping all access to the lucene index,
which has a queue where my Servlets or Actions put the documents to be
updated or added in.  There is a single instance of this class for the
whole web app, and a thread regularly waking up and processing the
elements of the queue.

Note the queue has to be threadsafe or has to be synchronized
externally. 

Since there is only one instance of this service class, it is the only
one who will ever write to the index (if the same index is not used by
other applications).

During re-indexing the thread regularly processing the queue will
be paused. After re-indexing it is started again, processing all pending 
changes from the queue. The re-indexing itself takes place in another
thread, which is started by quartz in my case.


hope this helps you somehow,

Jens


-- 
webit! Gesellschaft für neue Medien mbH          www.webit.de
Dipl.-Wirtschaftsingenieur Jens Krämer       kraemer@webit.de
Schnorrstraße 76                      Telefon +49 351 46766 0
D-01069 Dresden                      Telefax +49 351 46766 66

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


_________________________________________________________________________________________________________________________
Internet communications are not secure and therefore Oyster Partners Ltd does not accept legal
responsibility for the contents of this message. Any views or opinions presented are solely
those of the author and do not necessarily represent those of Oyster Partners Ltd.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message