lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Earwin Burrfoot <ear...@gmail.com>
Subject Re: Java caching of low-level index data?
Date Mon, 03 Aug 2009 12:50:04 GMT
> I'm curious if anyone has thought about (or even tried) caching the low-level index data
in Java, rather than
> in the OS.  For example, at the IndexInput level there could be an LRU cache of byte[]
blocks, similar to
> how a RDBMS caches index pages.  (Conveniently, BufferedIndexInput already reads in 1k
chunks.) You
> would reverse the advice above and instead make your JVM heap as large as possible (or
at least large
> enough to achieve a desired speed/space tradeoff).
I did something along these lines. It sucks. Having big Java heaps
ends you up with insane GC times. Loading GB-sized files into a bunch
of byte[1024] also wastes memory. Best bet by now is to rely on
mmap/file cache.

> I think swappiness is exactly the configuration that tells Linux just
> how happily it should swapp out application memory for IO cache vs
> other IO cache for new IO cache.
swappiness is roughly the percentage of free memory after which OS
starts searching for pages suitable for paging out. If set to low
values, OS wakes up in near-OOM conditions. If set to high values, as
soon as OS decides (according to some heuristics) that page is
eligible for page-out, it goes to disk.


-- 
Kirill Zakharenko/Кирилл Захаренко (earwin@gmail.com)
Home / Mobile: +7 (495) 683-567-4 / +7 (903) 5-888-423
ICQ: 104465785

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message