lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Rutherglen <jason.rutherg...@gmail.com>
Subject Re: Lucene memory usage
Date Thu, 11 Jun 2009 19:39:31 GMT
Maybe we can put together our requested IO operations and submit them for
inclusion in NIO Java 7?  http://openjdk.java.net/projects/nio/

On Thu, Jun 11, 2009 at 12:21 PM, Jason Rutherglen <
jason.rutherglen@gmail.com> wrote:

> Makes sense.
>
> Currently MMapDirectory doesn't write using mapped byte buffers,
> would the memory management of the OS behave differently if we
> were writing to the MMapped bytebuffers as opposed to writing to
> an RAF (like with FSDir)?
>
> > it's blind LRU approach is often a poor policy (eg for terms
> dict, where a binary search could easily suddenly need to visit
> a random rarely accessed page).
>
> Agreed it's not the best for termDict.
>
> > Well... locality is still important. Under the hood, mmap on a
> page miss must hit the disk.
>
> Maybe this is where MappedByteBuffer.load as Earwin has
> mantioned comes in handy?
>
> But yeah, we can't do anything with this unless we had a JNI
> library that interacts more directly with the IO system
> (allowing us to configure whether IO is cached etc), which
> perhaps exists or could exist in the future (or Java7?).
>
>
>
> On Thu, Jun 11, 2009 at 2:43 AM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>> On Wed, Jun 10, 2009 at 9:24 PM, Jason
>> Rutherglen<jason.rutherglen@gmail.com> wrote:
>> > I read over the LUCENE-1458 comments again. Interesting. I think
>> > the most compelling argument is that the various files we're
>> > normally loading into the heap are, after merging, in the IO
>> > cache. If we can simply reuse the IO cache rather then allocate
>> > a bunch of redundant arrays in heap, we could be better off? I
>> > think this is very compelling for field caches, delDocs, and
>> > bitsets that are tied to segments and loaded after each merge.
>>
>> The OS doesn't have enough information to "know" what data structures
>> are important to Lucene (must stay hot) and which are less so.  It's
>> blind LRU approach is often a poor policy (eg for terms dict, where a
>> binary search could easily suddenly need to visit a random rarely
>> accessed page).
>>
>> For example, after merging, all the segments we just *read* from will
>> also be hot, having flushed out other important pages from the IO
>> cache, which is very much not what we want to do.  From C, and per-OS,
>> you can inform the OS that it should not cache the bytes read from the
>> file, but from Java we just can't control that.
>>
>> > I think it's possible to write some basic benchmarks to test a
>> > byte[] BitVector vs.a MappedByteBuffer BitVector and see what
>> > happens.
>>
>> Yes, but this is challenging to test properly.  On systems with plenty
>> of RAM, the approaches should be similarly fast.  On systems starved
>> for RAM, both approaches should thrash miserably.  It's the cases in
>> between that we need to test for.
>>
>> > The other potentially interesting angle here is in regards to
>> > realtime updates, where we can implement a MMaped page type of
>> > system so blocks of this stuff can be updated in near realtime,
>> > directly in the MMaped space (similar to how in heap land with
>> > LUCENE-1526 we're looking at breaking up the byte[] into a
>> > byte[][]).
>>
>> But carrying such updates via RAM, like we do now for deletions,
>> should generally be more performant (you never have to put the changes
>> on disk).
>>
>> > Also if we assume data is MMaped I don't think it matters as much if
>> > the updates on disk are not in sequence? (Whereas today we try
>> > to keep all our files sequentially readable optimized). Of
>> > course I could be completely wrong. :)
>>
>> Well... locality is still important.  Under the hood, mmap on a page
>> miss must hit the disk.
>>
>> Mike
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>

Mime
View raw message