lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Rutherglen <jason.rutherg...@gmail.com>
Subject Re: madvise(ptr, len, MADV_SEQUENTIAL)
Date Wed, 17 Jun 2009 03:12:14 GMT
Perhaps we'd also like to request readahead be included in JDK7?

http://linux.die.net/man/2/readahead

On Tue, Jun 16, 2009 at 9:03 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:

> Hmm... posix_fadvise lets you do this with a file descriptor; this
> would be better for Lucene (per descriptor not per mapped region of
> RAM) since we could "advise" independent of which FSDir impl is in
> use...
>
> Mike
>
> On Tue, Jun 16, 2009 at 10:32 AM, Uwe Schindler<uwe@thetaphi.de> wrote:
> > But to use it, we should change MMapDirectory to also use the mapping
> when
> > writing to files. I thought about it, it is very simple to implement
> (just
> > copy the IndexInput and change all gets() to sets())
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >> -----Original Message-----
> >> From: Michael McCandless [mailto:lucene@mikemccandless.com]
> >> Sent: Tuesday, June 16, 2009 4:22 PM
> >> To: java-dev@lucene.apache.org
> >> Cc: Alan Bateman; nio-discuss@openjdk.java.net
> >> Subject: Re: madvise(ptr, len, MADV_SEQUENTIAL)
> >>
> >> Lucene could really make use of this method.  When a segment merge
> >> takes place, we can read & write many GB of data, which without
> >> madvise on many OSs would effectively flush the IO cache (thus hurting
> >> our search performance).
> >>
> >> Mike
> >>
> >> On Mon, Jun 15, 2009 at 6:01 PM, Jason
> >> Rutherglen<jason.rutherglen@gmail.com> wrote:
> >> > Thanks Alan.
> >> >
> >> > I cross posted this to the Lucene dev list where we are discussing
> using
> >> > madvise for minimizing unnecessary IO cache usage when merging
> segments
> >> > (where we really want the newly merged segments in the IO cache rather
> >> than
> >> > the old segment files).
> >> >
> >> > How would the advise method work?  Would there need to be a hint in
> the
> >> > FileChannel.map method?
> >> >
> >> > -J
> >> >
> >> > On Mon, Jun 15, 2009 at 12:36 AM, Alan Bateman <Alan.Bateman@sun.com>
> >> wrote:
> >> >>
> >> >> Jason Rutherglen wrote:
> >> >>>
> >> >>> Is there going to be a way to do this in the new Java IO APIs?
> >> >>
> >> >> Good question, as it has come up a few times and is needed for some
> >> >> important use-cases. A while back I looked into adding a
> >> >> MappedByteBuffer#advise method to allow the application provide hints
> >> on the
> >> >> expected usage but didn't complete it. We should probably look at
> this
> >> again
> >> >> for jdk7.
> >> >>
> >> >> -Alan.
> >> >>
> >> >
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>

Mime
View raw message