lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jong Kim <jong.luc...@gmail.com>
Subject Re: Lucene index on NFS
Date Tue, 02 Oct 2012 15:12:43 GMT
John,

Are you indicating that later Lucene releases might have a config setting
that can control the write I/O timeout? If so, do you happen to know where
it is or how to set it? I did quick Googling, but all I get back is the
write lock timeout which is set to one second by default.
Thanks
/Jong
On Tue, Oct 2, 2012 at 9:38 AM, Nader, John P <john.nader@cengage.com>wrote:

> We've been in production on Lucene over NFS for about 4 years now.  Though
> we've had performance issues related to NFS (similar to those mentioned on
> this thread), we've only seen some reliability issues.  Index writing I/O
> timeout exceptions are the primary issue.  We've addressed these by
> implementing retry logic.  This kept transactional consistency and avoided
> corruption.  I can't recall specifically where these timeouts occurred,
> but I do remember that on our version of Lucene at the time (3.0.2) the
> timeout was not configurable, and defaulted to 5 seconds.  Had it been
> configurable, we could have reduced how frequently we needed to rollback
> and retry.
>
> On another point, NFS performance can be greatly enhanced with servers
> that have extra memory for mapping the index files.  We found after
> initial warmup and loading of Lucene that queries performance very well
> since most of the data needed to execute them is cached locally.
>
> Also, keep in mind that local disk is not a free ride either.  This takes
> you down data replication.  You end up repeating indexing once per
> replica.  You also may have to move the indices around as you
> add/remove/restart nodes.  We are moving to this architecture with a new
> product, so I am just now starting to understand the trade-offs.
>
> Hope that helps.
>
> -John
>
>
>
>
> On 10/2/12 8:01 AM, "Jong Kim" <jong.lucene@gmail.com> wrote:
>
> >Thank you all for reply.
> >
> >So it soudns like it is a known fact that the performance would suffer
> >rather significantly when the index files are accessed over NFS. But how
> >about reliability and robustness (which seems even more important)? Isn't
> >there any increased possibility for intermittent errors such as index file
> >corruption (due to cache inconsistency, difference in delete semantics,
> >etc.) when using NFS? Has anyone run into such trouble? Or is it strictly
> >just a performance issue?
> >
> >/Jong
> >On Tue, Oct 2, 2012 at 5:17 AM, Paul Libbrecht <paul@hoplahup.net> wrote:
> >
> >> My experience in the Lucene 1.x times were a factor of at least four in
> >> writing to NFS and about two when reading from there. I'd discourage
> >>this
> >> as much as possible!
> >>
> >> (rsync is way more your friend for transporting and replication à la
> >>solr
> >> should also be considered)
> >>
> >> paul
> >>
> >>
> >> Le 2 oct. 2012 à 11:10, Ian Lea a écrit :
> >>
> >> > You'll certainly need to factor in the performance of NFS versus local
> >> disks.
> >> >
> >> > My experience is that smallish low activity indexes work just fine on
> >> > NFS, but large high activity indexes are not so good, particularly if
> >> > you have a lot of modifications to the index.
> >> >
> >> > You may want to install a custom IndexDeletionPolicy.  See the
> >> > javadocs for details with specific reference to NFS.
> >> >
> >> >
> >> > --
> >> > Ian.
> >> >
> >> > On Tue, Oct 2, 2012 at 3:21 AM, Vitaly Funstein <vfunstein@gmail.com>
> >> wrote:
> >> >> How tolerant is your project of decreased search and indexing
> >> performance?
> >> >> You could probably write a simple test that compares search and write
> >> >> speeds of local and NFS-mounted indexes and make the decision based
> >>on
> >> the
> >> >> results.
> >> >>
> >> >> On Mon, Oct 1, 2012 at 3:06 PM, Jong Kim <jong.lucene@gmail.com>
> >>wrote:
> >> >>
> >> >>> Hi,
> >> >>>
> >> >>> According to the Lucene In Action (Second Edition), the section
> >>2.11.2
> >> >>> "Accessing an index over a remote file system" explains that there
> >>are
> >> >>> issues related to accessing a Lucene index across remote file system
> >> >>> including NFS.
> >> >>>
> >> >>> I'm particuarly interested in NFS compatibility, and wondering
if
> >> there has
> >> >>> been any work done to solve or mitigate this problem. Has this
issue
> >> been
> >> >>> addressed? If not, are there some reliable work-arounds that make
> >>this
> >> >>> possible at the expense of some sacrifice in other areas?
> >> >>>
> >> >>> Any information would be greatly appreciated, since my project
> >>heavily
> >> >>> depends on the feasibility of this.
> >> >>>
> >> >>> Thanks
> >> >>> /Jong
> >> >>>
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >> >
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message