Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (hermes.apache.org: local policy)
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable
Subject: RE: Strategies for updating indexes.
Date: Tue, 5 Apr 2005 11:24:15 +0200
Message-ID: <00ADB1D1C6D9434989D51BA9E55FAE0E0242559B@de010367.de.ina.com>
Thread-Topic: Strategies for updating indexes.
Thread-Index: AcU5smdMB5egXd75QjmAezWA3cABSQACz4ew
From: "Nestel, Frank  IZ/HZA-IOL" <nestefan@de.ina.com>
To: <java-user@lucene.apache.org>

Hi,

we are using a very cautious method for batch upating.=20

We have long (hours) running updates on our index, but=20
complete reindexing would even be longer (days). But I
guess our strategy could be scaled down to hours or even
less.

So what we do is, we keep two instances
of the index. There is a file which contains a link to the
currently used index for reading. The search application
accesses the index through a thin transparent API which
whatches this file and can switch from one search request
to the next, but does keep the IndexReader open as long
as no switching is need.

When we reindex, we duplicate the "read" instance of the=20
index and then do an selective update on this copied duplicate.=20
Note we need some HD space and time for this. After indexing
there is some checking and comparison between old and new
instance of the index. Only if this looks successful we=20
toggle the above "pointer" file and the transparent API switches
to the new index. In case human recognition still finds the
checked index bad, we can still switch back to the second
last presumably "good" index.=20

Cheers,
Frank

>-----Original Message-----
>From: Lee Turner [mailto:Lee.Turner@oyster.com]=20
>Sent: Tuesday, April 05, 2005 9:38 AM
>To: java-user@lucene.apache.org
>Subject: Strategies for updating indexes.
>
>
>Hi
>
>=20
>
>I was wondering whether anyone has any experience of=20
>multithreaded updates to indexes.  I the web app I am working=20
>on there are additions, updates and deletes that need to=20
>happen to the index throughout the runtime of the application.=20
> Also, the application is run in a cluster with each app=20
>server having its own index.  This means that periodically=20
>each app server is going to have to go through a re-indexing=20
>process to make sure that its index has all the changes from=20
>the other app servers in it.  This process can take a few=20
>seconds so if another update to the index occurs at this time=20
>it will need to be queued in some way to make sure it happens=20
>after the re-indexing.
>
>=20
>
>I was just wondering if anyone had any pointers for doing this=20
>kind of thing.  Any help would be gratefully appreciated.
>
>=20
>
>Many thanks
>
>Lee
>
>=20
>
>=20
>
>Lee Turner | Java Developer | Oyster Partners=20
>
>D. +44 (0)20 74461229=20
>T. +44 (0)20 7446 7500=20
>
>www.oyster.com
>
>=20
>
>
>_______________________________________________________________
>__________________________________________________________
>Internet communications are not secure and therefore Oyster=20
>Partners Ltd does not accept legal responsibility for the=20
>contents of this message. Any views or opinions presented are=20
>solely those of the author and do not necessarily represent=20
>those of Oyster Partners Ltd.
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org