lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andy Liu" <andyliu1...@gmail.com>
Subject Re: Index updates between machines
Date Tue, 03 Apr 2007 19:55:50 GMT
Sounds like you might have an I/O issue.  If you have multiple partitions /
disks on the searching server you can search from one partition and copy to
another and alternate.  If you're using RAID different RAID levels are
optimized for simultaneous reads and writes.

If you have a 3rd machine you can load balance 2 search servers and take one
out of the cluster when the index is being copied.  Alternatively, if it's
possible, you can copy the index at an offpeak hour.

Andy

On 4/3/07, Otis Gospodnetic <otis_gospodnetic@yahoo.com> wrote:
>
> How fast are your disks?  Perhaps they are having trouble keeping up with
> simultaneous searches and massive file copying.
>
> Otis
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Simpy -- http://www.simpy.com/  -  Tag  -  Search  -  Share
>
> ----- Original Message ----
> From: Chun Wei Ho <cwho.work@gmail.com>
> To: java-user@lucene.apache.org
> Sent: Tuesday, April 3, 2007 10:40:16 AM
> Subject: Index updates between machines
>
> We are running a search service on the internet using two machines. We
> have a crawler machine which crawls the web and merges new documents
> found into the Lucene index. We have a searcher machine which allows
> users to perform searches on the Lucene index.
>
> Periodically, we would copy the newest version of the index from the
> crawler machine over to the searcher machine (via copy over a NFS
> mount). The searcher would then detect the new version, close the old
> index, open the new index and resume the search service.
>
> As the index have been growing in size, we have been noticing that the
> search response time on the searcher machine increases drastically
> when an index (about 15GB) is being copied from the crawler to the
> searcher. Both machines run Fedora Core 4 and are on a gbps lan.
>
> We've tried a number of ways to reduce the impact of the copy over NFS
> on searching performance, such as "nice"ing the copy process, but to
> no avail. I wonder if anyone is running a lucene search service over a
> similar architecture and how you are managing the updates to the
> lucene index.
>
> Thanks!
>
> Regards,
> CW
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message