lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Medinets" <medi...@mtolive.com>
Subject Re: commercial websites powered by Lucene?
Date Tue, 24 Jun 2003 13:24:25 GMT
----- Original Message -----
From: "Chris Miller" <chris_overseas@hotmail.com>
> Thanks David, that's about what I figured. Of course if the servers are
> pulling the information then a central holding table that contains only
new
> data doesn't make much sense anymore. Instead I guess the easiest approach
> would be to have a central table that contains the entire dataset

The following commentary might have no bearing on Lucene nor relevance with
today's technology, but I feel garrulous this morning.

Each pulling server did a three-step dance when updating. First, the central
server (Oracle) was polled to get the latest data (actually we sucked it all
because there were only 30,000 records). A text file was created (format is
unimportant, use the easiest for your application). Then that text file was
read to update the local datastore.

The advantage of this rigamarole was to allow the servers to fail and be
restored without needing to poll the central server. We had 400 servers in
the cluster. And at times, many of them would be fail (this was in 1999,
don't be critical!). If many systems pulled data from the central server,
the process would slow down. Which started another round of failures. To
avoid that vicious circle of failure all of the systems could reboot
independently.

David Medinets
http://www.codebits.com




---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message