lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Darren Govoni <dar...@ontrenet.com>
Subject Re: possible to read index into memory?
Date Wed, 13 Aug 2008 11:42:39 GMT
Hoss,
   Thank you for the detailed response. What I found weird was it
seemed to take 0.09 seconds to create a RAMDirectory off a 17MB index.
Suspiciously fast, but ok.

Yet, when I do a simple fuzzy search on a single field 

"word: someword~0.76"

It was taking .35 seconds. That's a very very long time all things
considered. I understand about the OS paging and such but in
doing some variations of this to "throw the OS off", I still saw
no difference between on-disk and RAM times. But despite that, the
times are really slow. 

Any ideas?

thanks again,
Darren

On Tue, 2008-08-12 at 19:55 -0700, Chris Hostetter wrote:
> : On one index, I am seeing no speed change when flipping between
> : RAMDirectory IndexSearcher and file system version.
> 
> that is probably because even if you just use an FSDirectory, your OS will 
> cache the disk "pages" in RAM for you -- all using a RAMDirectory does for 
> you is garuntee that the entire index is copied into the heap you allocate 
> for your JVM.  If you've got 16GB or RAM, and a 5GB index, and you 
> allocated 12GB of RAM to the JVM and read your index into a RAMDirectory, 
> your index will always be in RAM, no matter what other processes do on 
> your machine.
> 
> If instead you only allocate 6GB of RAM to the JVM, and nothing else is 
> using up the rest of your RAM, the OS has plenty to load the whole index 
> into RAM as part of the filesystem cache once you use it -- but if another 
> process comes along and really needs that RAM (or if something reads a lot 
> of other pages of disk) your index might get bumped from the filesystem 
> cache, and the next few reads could be slow.
> 
> : Creating the RAMDirectory from the on-disk index only takes 0.09
> : seconds. It appears it is not loading the data into memory, but maybe
> : just the file names of the index?
> 
> passing an FSDIrectory to the constructor of a RAMDIrectory uses the 
> Directory.copy() method whose source is fairly straight forward and easy 
> to read -- unless your index is ginormous it's not suprising that it's 
> "fast" particularly if it's already in the filesystem cache.
> 
> 
> 
> 
> -Hoss
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message