lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Hostetter <hossman_luc...@fucit.org>
Subject Re: possible to read index into memory?
Date Wed, 13 Aug 2008 02:55:28 GMT

: On one index, I am seeing no speed change when flipping between
: RAMDirectory IndexSearcher and file system version.

that is probably because even if you just use an FSDirectory, your OS will 
cache the disk "pages" in RAM for you -- all using a RAMDirectory does for 
you is garuntee that the entire index is copied into the heap you allocate 
for your JVM.  If you've got 16GB or RAM, and a 5GB index, and you 
allocated 12GB of RAM to the JVM and read your index into a RAMDirectory, 
your index will always be in RAM, no matter what other processes do on 
your machine.

If instead you only allocate 6GB of RAM to the JVM, and nothing else is 
using up the rest of your RAM, the OS has plenty to load the whole index 
into RAM as part of the filesystem cache once you use it -- but if another 
process comes along and really needs that RAM (or if something reads a lot 
of other pages of disk) your index might get bumped from the filesystem 
cache, and the next few reads could be slow.

: Creating the RAMDirectory from the on-disk index only takes 0.09
: seconds. It appears it is not loading the data into memory, but maybe
: just the file names of the index?

passing an FSDIrectory to the constructor of a RAMDIrectory uses the 
Directory.copy() method whose source is fairly straight forward and easy 
to read -- unless your index is ginormous it's not suprising that it's 
"fast" particularly if it's already in the filesystem cache.




-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message