lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Rutherglen <jason.rutherg...@gmail.com>
Subject Re: madvise(ptr, len, MADV_SEQUENTIAL)
Date Wed, 17 Jun 2009 03:20:50 GMT
Sorry, not portable, but POSIX_FADV_WILLNEED is which can be used with
posix_fadvise.

On Tue, Jun 16, 2009 at 8:12 PM, Jason Rutherglen <
jason.rutherglen@gmail.com> wrote:

> Perhaps we'd also like to request readahead be included in JDK7?
>
> http://linux.die.net/man/2/readahead
>
> On Tue, Jun 16, 2009 at 9:03 AM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>> Hmm... posix_fadvise lets you do this with a file descriptor; this
>> would be better for Lucene (per descriptor not per mapped region of
>> RAM) since we could "advise" independent of which FSDir impl is in
>> use...
>>
>> Mike
>>
>> On Tue, Jun 16, 2009 at 10:32 AM, Uwe Schindler<uwe@thetaphi.de> wrote:
>> > But to use it, we should change MMapDirectory to also use the mapping
>> when
>> > writing to files. I thought about it, it is very simple to implement
>> (just
>> > copy the IndexInput and change all gets() to sets())
>> >
>> > -----
>> > Uwe Schindler
>> > H.-H.-Meier-Allee 63, D-28213 Bremen
>> > http://www.thetaphi.de
>> > eMail: uwe@thetaphi.de
>> >
>> >> -----Original Message-----
>> >> From: Michael McCandless [mailto:lucene@mikemccandless.com]
>> >> Sent: Tuesday, June 16, 2009 4:22 PM
>> >> To: java-dev@lucene.apache.org
>> >> Cc: Alan Bateman; nio-discuss@openjdk.java.net
>> >> Subject: Re: madvise(ptr, len, MADV_SEQUENTIAL)
>> >>
>> >> Lucene could really make use of this method.  When a segment merge
>> >> takes place, we can read & write many GB of data, which without
>> >> madvise on many OSs would effectively flush the IO cache (thus hurting
>> >> our search performance).
>> >>
>> >> Mike
>> >>
>> >> On Mon, Jun 15, 2009 at 6:01 PM, Jason
>> >> Rutherglen<jason.rutherglen@gmail.com> wrote:
>> >> > Thanks Alan.
>> >> >
>> >> > I cross posted this to the Lucene dev list where we are discussing
>> using
>> >> > madvise for minimizing unnecessary IO cache usage when merging
>> segments
>> >> > (where we really want the newly merged segments in the IO cache
>> rather
>> >> than
>> >> > the old segment files).
>> >> >
>> >> > How would the advise method work?  Would there need to be a hint in
>> the
>> >> > FileChannel.map method?
>> >> >
>> >> > -J
>> >> >
>> >> > On Mon, Jun 15, 2009 at 12:36 AM, Alan Bateman <Alan.Bateman@sun.com
>> >
>> >> wrote:
>> >> >>
>> >> >> Jason Rutherglen wrote:
>> >> >>>
>> >> >>> Is there going to be a way to do this in the new Java IO APIs?
>> >> >>
>> >> >> Good question, as it has come up a few times and is needed for
some
>> >> >> important use-cases. A while back I looked into adding a
>> >> >> MappedByteBuffer#advise method to allow the application provide
>> hints
>> >> on the
>> >> >> expected usage but didn't complete it. We should probably look
at
>> this
>> >> again
>> >> >> for jdk7.
>> >> >>
>> >> >> -Alan.
>> >> >>
>> >> >
>> >> >
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
>> >
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> > For additional commands, e-mail: java-dev-help@lucene.apache.org
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>

Mime
View raw message