lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <>
Subject Re: Lucene memory usage
Date Fri, 12 Jun 2009 10:17:12 GMT
On Thu, Jun 11, 2009 at 4:30 PM, Jason
Rutherglen<> wrote:
>> Yes please post feature requests to Sun ;)
> I signed up for

Looks like a fun list ;)

>> But I think in the short term Lucene will have to drop to
> native code to tell OS not to cache bytes read by segment
> merging...
> LUCENE-1121 uses transferTo which presumably doesn't run bytes
> through the IO cache? Granted it's slower on most platforms, but
> could this be fixed in future Java releases?

I fear transferTo may in fact burn through the IO cache... or, at
least, when I tested it in the past, on multi-GB files, I saw vmem
used by the process grow enormously... I think OS was choosing to use
massive chunks of RAM as the intermediate buffer (slurping lots of
bytes in, and then writing lots of bytes out), to minimize seeking.
We got mixed results on the performance gains on that issue, and some
(Windows Server 2003) were hideously slower.  I think the OS may have
been evicting dirty pages, or swapping app core out, or something, in
order to free up RAM.

We really need better control on what the OS does w/ IO and our RAM,
but we lack that from Java.  EG if I had the option I'd like to
provide the option to Lucene users to pin certain data structures
(like the terms index) to ensure the OS never swaps it out.  But we
just can't do that from Java...


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message