lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jong Kim <jong.luc...@gmail.com>
Subject Re: Lucene index on NFS
Date Tue, 02 Oct 2012 13:29:17 GMT
The setup is I have a home-grown server process that has exclusive access
to the index files. All reads and writes are done through this server. No
other process is reading the same index files whether it's local or over
NFS.
/Jong
On Tue, Oct 2, 2012 at 8:56 AM, Ian Lea <ian.lea@gmail.com> wrote:

> I agree that reliability/corruption is not an issue.
>
> I would also put it that performance is likely to suffer, but that's
> not certain.  A fast disk mounted over NFS can be quicker than a slow
> local disk.  And how much do you care about performance?  Maybe it
> would be fast enough over NFS to make the ease of deployment balance
> out lesser speed.
>
> What's the setup here?  Will you be writing to an index on local disk
> of server A and reading it, over NFS, from server B (and C and ...) or
> what?
>
> --
> Ian.
>
>
> On Tue, Oct 2, 2012 at 1:45 PM, Paul Libbrecht <paul@hoplahup.net> wrote:
> > I doubt NFS is an unreliable file-system.
> > Lucene uses normal random access to files and this has no reason to be
> unreliable unless bad things such as network drops happen (in which case
> you'd get direct failures or  timeouts rather than corruption). I've seen
> fairly large infrastructures being based on NFS and corruption is something
> I've never heard about.
> >
> > Note: no concurrent access to a lucene index, right?
> >
> > Paul
> >
> >
> > Le 2 oct. 2012 à 14:01, Jong Kim a écrit :
> >
> >> Thank you all for reply.
> >>
> >> So it soudns like it is a known fact that the performance would suffer
> >> rather significantly when the index files are accessed over NFS. But how
> >> about reliability and robustness (which seems even more important)?
> Isn't
> >> there any increased possibility for intermittent errors such as index
> file
> >> corruption (due to cache inconsistency, difference in delete semantics,
> >> etc.) when using NFS? Has anyone run into such trouble? Or is it
> strictly
> >> just a performance issue?
> >>
> >> /Jong
> >> On Tue, Oct 2, 2012 at 5:17 AM, Paul Libbrecht <paul@hoplahup.net>
> wrote:
> >>
> >>> My experience in the Lucene 1.x times were a factor of at least four in
> >>> writing to NFS and about two when reading from there. I'd discourage
> this
> >>> as much as possible!
> >>>
> >>> (rsync is way more your friend for transporting and replication à la
> solr
> >>> should also be considered)
> >>>
> >>> paul
> >>>
> >>>
> >>> Le 2 oct. 2012 à 11:10, Ian Lea a écrit :
> >>>
> >>>> You'll certainly need to factor in the performance of NFS versus local
> >>> disks.
> >>>>
> >>>> My experience is that smallish low activity indexes work just fine on
> >>>> NFS, but large high activity indexes are not so good, particularly if
> >>>> you have a lot of modifications to the index.
> >>>>
> >>>> You may want to install a custom IndexDeletionPolicy.  See the
> >>>> javadocs for details with specific reference to NFS.
> >>>>
> >>>>
> >>>> --
> >>>> Ian.
> >>>>
> >>>> On Tue, Oct 2, 2012 at 3:21 AM, Vitaly Funstein <vfunstein@gmail.com>
> >>> wrote:
> >>>>> How tolerant is your project of decreased search and indexing
> >>> performance?
> >>>>> You could probably write a simple test that compares search and
write
> >>>>> speeds of local and NFS-mounted indexes and make the decision based
> on
> >>> the
> >>>>> results.
> >>>>>
> >>>>> On Mon, Oct 1, 2012 at 3:06 PM, Jong Kim <jong.lucene@gmail.com>
> wrote:
> >>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> According to the Lucene In Action (Second Edition), the section
> 2.11.2
> >>>>>> "Accessing an index over a remote file system" explains that
there
> are
> >>>>>> issues related to accessing a Lucene index across remote file
system
> >>>>>> including NFS.
> >>>>>>
> >>>>>> I'm particuarly interested in NFS compatibility, and wondering
if
> >>> there has
> >>>>>> been any work done to solve or mitigate this problem. Has this
issue
> >>> been
> >>>>>> addressed? If not, are there some reliable work-arounds that
make
> this
> >>>>>> possible at the expense of some sacrifice in other areas?
> >>>>>>
> >>>>>> Any information would be greatly appreciated, since my project
> heavily
> >>>>>> depends on the feasibility of this.
> >>>>>>
> >>>>>> Thanks
> >>>>>> /Jong
> >>>>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>>
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>
> >>>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message