lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jong Kim <jong.luc...@gmail.com>
Subject Re: Lucene index on NFS
Date Tue, 02 Oct 2012 15:19:58 GMT
OK, so it sounds like I'm hearing that

(a) Accessing index files over NFS from a "single" physical process on
a single computer is safe and can be made to work.

(b) Accessing index files over NFS from "multiple" processes/machines might
be problematic

(c) In all cases, the performance would be lesser over NFS
Thanks
/Jong
On Tue, Oct 2, 2012 at 9:36 AM, Tommaso Teofili
<tommaso.teofili@gmail.com>wrote:

> Ok, that saves you from concurrency issue, but in my experience is just
> much slower than local file system, so still NFS can be used but with some
> tradeoff on performance.
>
> My 2 cents,
> Tommaso
>
> 2012/10/2 Jong Kim <jong.lucene@gmail.com>
>
> > The setup is I have a home-grown server process that has exclusive access
> > to the index files. All reads and writes are done through this server. No
> > other process is reading the same index files whether it's local or over
> > NFS.
> > /Jong
> > On Tue, Oct 2, 2012 at 8:56 AM, Ian Lea <ian.lea@gmail.com> wrote:
> >
> > > I agree that reliability/corruption is not an issue.
> > >
> > > I would also put it that performance is likely to suffer, but that's
> > > not certain.  A fast disk mounted over NFS can be quicker than a slow
> > > local disk.  And how much do you care about performance?  Maybe it
> > > would be fast enough over NFS to make the ease of deployment balance
> > > out lesser speed.
> > >
> > > What's the setup here?  Will you be writing to an index on local disk
> > > of server A and reading it, over NFS, from server B (and C and ...) or
> > > what?
> > >
> > > --
> > > Ian.
> > >
> > >
> > > On Tue, Oct 2, 2012 at 1:45 PM, Paul Libbrecht <paul@hoplahup.net>
> > wrote:
> > > > I doubt NFS is an unreliable file-system.
> > > > Lucene uses normal random access to files and this has no reason to
> be
> > > unreliable unless bad things such as network drops happen (in which
> case
> > > you'd get direct failures or  timeouts rather than corruption). I've
> seen
> > > fairly large infrastructures being based on NFS and corruption is
> > something
> > > I've never heard about.
> > > >
> > > > Note: no concurrent access to a lucene index, right?
> > > >
> > > > Paul
> > > >
> > > >
> > > > Le 2 oct. 2012 à 14:01, Jong Kim a écrit :
> > > >
> > > >> Thank you all for reply.
> > > >>
> > > >> So it soudns like it is a known fact that the performance would
> suffer
> > > >> rather significantly when the index files are accessed over NFS. But
> > how
> > > >> about reliability and robustness (which seems even more important)?
> > > Isn't
> > > >> there any increased possibility for intermittent errors such as
> index
> > > file
> > > >> corruption (due to cache inconsistency, difference in delete
> > semantics,
> > > >> etc.) when using NFS? Has anyone run into such trouble? Or is it
> > > strictly
> > > >> just a performance issue?
> > > >>
> > > >> /Jong
> > > >> On Tue, Oct 2, 2012 at 5:17 AM, Paul Libbrecht <paul@hoplahup.net>
> > > wrote:
> > > >>
> > > >>> My experience in the Lucene 1.x times were a factor of at least
> four
> > in
> > > >>> writing to NFS and about two when reading from there. I'd
> discourage
> > > this
> > > >>> as much as possible!
> > > >>>
> > > >>> (rsync is way more your friend for transporting and replication
à
> la
> > > solr
> > > >>> should also be considered)
> > > >>>
> > > >>> paul
> > > >>>
> > > >>>
> > > >>> Le 2 oct. 2012 à 11:10, Ian Lea a écrit :
> > > >>>
> > > >>>> You'll certainly need to factor in the performance of NFS
versus
> > local
> > > >>> disks.
> > > >>>>
> > > >>>> My experience is that smallish low activity indexes work just
fine
> > on
> > > >>>> NFS, but large high activity indexes are not so good, particularly
> > if
> > > >>>> you have a lot of modifications to the index.
> > > >>>>
> > > >>>> You may want to install a custom IndexDeletionPolicy.  See
the
> > > >>>> javadocs for details with specific reference to NFS.
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> Ian.
> > > >>>>
> > > >>>> On Tue, Oct 2, 2012 at 3:21 AM, Vitaly Funstein <
> > vfunstein@gmail.com>
> > > >>> wrote:
> > > >>>>> How tolerant is your project of decreased search and indexing
> > > >>> performance?
> > > >>>>> You could probably write a simple test that compares search
and
> > write
> > > >>>>> speeds of local and NFS-mounted indexes and make the decision
> based
> > > on
> > > >>> the
> > > >>>>> results.
> > > >>>>>
> > > >>>>> On Mon, Oct 1, 2012 at 3:06 PM, Jong Kim <jong.lucene@gmail.com>
> > > wrote:
> > > >>>>>
> > > >>>>>> Hi,
> > > >>>>>>
> > > >>>>>> According to the Lucene In Action (Second Edition),
the section
> > > 2.11.2
> > > >>>>>> "Accessing an index over a remote file system" explains
that
> there
> > > are
> > > >>>>>> issues related to accessing a Lucene index across
remote file
> > system
> > > >>>>>> including NFS.
> > > >>>>>>
> > > >>>>>> I'm particuarly interested in NFS compatibility, and
wondering
> if
> > > >>> there has
> > > >>>>>> been any work done to solve or mitigate this problem.
Has this
> > issue
> > > >>> been
> > > >>>>>> addressed? If not, are there some reliable work-arounds
that
> make
> > > this
> > > >>>>>> possible at the expense of some sacrifice in other
areas?
> > > >>>>>>
> > > >>>>>> Any information would be greatly appreciated, since
my project
> > > heavily
> > > >>>>>> depends on the feasibility of this.
> > > >>>>>>
> > > >>>>>> Thanks
> > > >>>>>> /Jong
> > > >>>>>>
> > > >>>>
> > > >>>>
> > ---------------------------------------------------------------------
> > > >>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > >>>> For additional commands, e-mail: java-user-help@lucene.apache.org
> > > >>>>
> > > >>>
> > > >>>
> > > >>>
> ---------------------------------------------------------------------
> > > >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > >>> For additional commands, e-mail: java-user-help@lucene.apache.org
> > > >>>
> > > >>>
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message