lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nigel <>
Subject Re: Java caching of low-level index data?
Date Thu, 23 Jul 2009 14:03:40 GMT
I think this approach would make sense only in certain situations.  If
you're running different things with different memory requirements on the
same server, then the OS is probably the most efficient judge of what needs
to be in RAM.  However, if (like us) the server is running a big Java app
using Lucene and nothing else, then maintaining your own cache seems useful,
for the reasons I mentioned earlier.

Mike, the question you raise is whether (or to what degree) the OS will swap
out app memory in favor of IO cache.  I don't know anything about how the
Linux kernel makes those decisions, but I guess I had hoped that (regardless
of the swappiness setting) it would be less likely to swap out application
memory for IO, than it would be to replace some cached IO data with some
different cached IO data.  The latter case is what kills Lucene performance
when you've got a lot of index data in the IO cache and a file copy or some
other operation replaces it all with something else: the OS has no way of
knowing that some IO cache is more desirable long-term than other IO cache.
The former case (swapping app for IO cache) makes sense, I suppose, if the
app memory hasn't been used in a long time, but with an LRU cache you should
be hitting those pages pretty frequently by definition.  But if it does swap
out your Java cache for something else, you're probably no worse off than
before, right?  In this case you have to hit the disk to fault in the
paged-out cache; in the original case you have to hit the disk to read the
index data that's not in IO cache.

Anyway, the interaction between these things (virtual memory, IO cache,
disk, JVM, garbage collection, etc.) are complex and so the optimal
configuration is very usage-dependent.  The current Lucene behavior seems to
be the most flexible.  When/if I get a chance to try the Java caching for
our situation I'll report the results.

On Wed, Jul 22, 2009 at 12:37 PM, Michael McCandless <> wrote:

> I think it's a neat idea!
> But you are in fact fighting the OS so I'm not sure how well this'll
> work in practice.
> EG the OS will happily swap out pages from your process if it thinks
> you're not using them, so it'd easily swap out your cache in favor of
> its own IO cache (this is the "swappiness" configuration on Linux),
> which would then kill performance (take a page hit when you finally
> did need to use your cache).  In C (possibly requiring root) you could
> wire the pages, but we can't do that from javaland, so it's already
> not a fair fight.
> Mike
> > On Wed, Jul 22, 2009 at 6:21 PM, Nigel <> wrote:
> >>
> >> In discussions of Lucene search performance, the importance of OS
> caching
> >> of index data is frequently mentioned.  The typical recommendation is to
> >> keep plenty of unallocated RAM available (e.g. don't gobble it all up
> with
> >> your JVM heap) and try to avoid large I/O operations that would purge
> the OS
> >> cache.
> >>
> >> I'm curious if anyone has thought about (or even tried) caching the
> >> low-level index data in Java, rather than in the OS.  For example, at
> the
> >> IndexInput level there could be an LRU cache of byte[] blocks, similar
> to
> >> how a RDBMS caches index pages.  (Conveniently, BufferedIndexInput
> already
> >> reads in 1k chunks.) You would reverse the advice above and instead make
> >> your JVM heap as large as possible (or at least large enough to achieve
> a
> >> desired speed/space tradeoff).
> >>
> >> This approach seems like it would have some advantages:
> >>
> >> - Explicit control over how much you want cached (adjust your JVM heap
> and
> >> cache settings as desired)
> >> - Cached index data won't be purged by the OS doing other things
> >> - Index warming might be faster, or at least more predictable
> >>
> >> The obvious disadvantage for some situations is that more RAM would now
> be
> >> tied up by the JVM, rather than managed dynamically by the OS.
> >>
> >> Any thoughts?  It seems like this would be pretty easy to implement
> >> (subclass FSDirectory, return subclass of FSIndexInput that checks the
> cache
> >> before reading, cache keyed on filename + position), but maybe I'm
> >> oversimplifying, and for that matter a similar implementation may
> already
> >> exist somewhere for all I know.

View raw message