lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nigel <>
Subject Re: Java caching of low-level index data?
Date Wed, 05 Aug 2009 15:36:42 GMT
Thanks, that's good feedback!


On Mon, Aug 3, 2009 at 8:50 AM, Earwin Burrfoot <> wrote:

> > I'm curious if anyone has thought about (or even tried) caching the
> low-level index data in Java, rather than
> > in the OS.  For example, at the IndexInput level there could be an LRU
> cache of byte[] blocks, similar to
> > how a RDBMS caches index pages.  (Conveniently, BufferedIndexInput
> already reads in 1k chunks.) You
> > would reverse the advice above and instead make your JVM heap as large as
> possible (or at least large
> > enough to achieve a desired speed/space tradeoff).
> I did something along these lines. It sucks. Having big Java heaps
> ends you up with insane GC times. Loading GB-sized files into a bunch
> of byte[1024] also wastes memory. Best bet by now is to rely on
> mmap/file cache.
> > I think swappiness is exactly the configuration that tells Linux just
> > how happily it should swapp out application memory for IO cache vs
> > other IO cache for new IO cache.
> swappiness is roughly the percentage of free memory after which OS
> starts searching for pages suitable for paging out. If set to low
> values, OS wakes up in near-OOM conditions. If set to high values, as
> soon as OS decides (according to some heuristics) that page is
> eligible for page-out, it goes to disk.

View raw message