lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Earwin Burrfoot <ear...@gmail.com>
Subject Re: madvise(ptr, len, MADV_SEQUENTIAL)
Date Tue, 16 Jun 2009 15:48:06 GMT
Except, you don't know the size of the file to be written upfront.
One probable solution is to map output file in pages. As a
complementary solution you can map a huge area of the file, and hope
few real memory is allocated by OS unless you actually write all over
that area.
Dunno. The idea of using mmapped write has stopped looking interesting to me.

On Tue, Jun 16, 2009 at 18:32, Uwe Schindler<uwe@thetaphi.de> wrote:
> But to use it, we should change MMapDirectory to also use the mapping when
> writing to files. I thought about it, it is very simple to implement (just
> copy the IndexInput and change all gets() to sets())
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>> -----Original Message-----
>> From: Michael McCandless [mailto:lucene@mikemccandless.com]
>> Sent: Tuesday, June 16, 2009 4:22 PM
>> To: java-dev@lucene.apache.org
>> Cc: Alan Bateman; nio-discuss@openjdk.java.net
>> Subject: Re: madvise(ptr, len, MADV_SEQUENTIAL)
>>
>> Lucene could really make use of this method.  When a segment merge
>> takes place, we can read & write many GB of data, which without
>> madvise on many OSs would effectively flush the IO cache (thus hurting
>> our search performance).
>>
>> Mike
>>
>> On Mon, Jun 15, 2009 at 6:01 PM, Jason
>> Rutherglen<jason.rutherglen@gmail.com> wrote:
>> > Thanks Alan.
>> >
>> > I cross posted this to the Lucene dev list where we are discussing using
>> > madvise for minimizing unnecessary IO cache usage when merging segments
>> > (where we really want the newly merged segments in the IO cache rather
>> than
>> > the old segment files).
>> >
>> > How would the advise method work?  Would there need to be a hint in the
>> > FileChannel.map method?
>> >
>> > -J
>> >
>> > On Mon, Jun 15, 2009 at 12:36 AM, Alan Bateman <Alan.Bateman@sun.com>
>> wrote:
>> >>
>> >> Jason Rutherglen wrote:
>> >>>
>> >>> Is there going to be a way to do this in the new Java IO APIs?
>> >>
>> >> Good question, as it has come up a few times and is needed for some
>> >> important use-cases. A while back I looked into adding a
>> >> MappedByteBuffer#advise method to allow the application provide hints
>> on the
>> >> expected usage but didn't complete it. We should probably look at this
>> again
>> >> for jdk7.
>> >>
>> >> -Alan.
>> >>
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>



-- 
Kirill Zakharenko/Кирилл Захаренко (earwin@gmail.com)
Home / Mobile: +7 (495) 683-567-4 / +7 (903) 5-888-423
ICQ: 104465785

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message