lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kay Kay <kaykay.uni...@gmail.com>
Subject Re: Reloading RAM Directory from updated FS Directory
Date Tue, 09 Jun 2009 22:06:29 GMT
Have you checked out solr project that provides a service on top of 
Lucene + caching / warming up facilities similar to what you need.

The IndexReaders are expensive ( and are the underlying data source for 
a given IndexSearcher ) in terms of time and resources , when being 
opened / created anew.

So - having 2 live instances of IndexReaders concurrently might be an 
issue to begin with.

Lucene (IndexSearchers / IndexReaders) has the notion of cache as well 
so you need to check if you really want a 100% replication of 
RAMDirectory / FSDirectory as well concurrently in memory.  Have you 
tested with the FieldCache policies before moving onto the RAMDirectory 
based backup solution.

--
  Karthik.


Diamond, Greg wrote:
> Hi All -
>
> What is the best way to load a RAM Directory from a FS Directory, and periodically reload
the RAM Directory to pick up new documents?
>
> The scenario I have is I create several large directories which I create to a file system,
then load them into ram for faster searching.
> They takes several hours to create, so i want to retain a file copy so in the event of
a service/server crash or reboot, they can load again
> in a few seconds.  Once a day I append or recreate the FS Directories with new entries,
and then reload the RAM Directories to pick up the
> new entries.
>
> Problem is, what I am doing seems to cause a memory leak.  The directories take ~12 GB
of RAM to load, but bloats
> to 24 GB (the -Xmx setting) after a few reloads and stays there.
>
> 	// Map to hold single instance of the IndexSearcher, 1 per RAM Directory, to be reused
across requests.
> 	private static final Map<IndexName, IndexSearcher> searchers = new HashMap<IndexName,
IndexSearcher>();
> 	
> 	// Creates the IndexSearcher and stores it in the static map.
> 	public static void load(IndexName index, ...) {
> 		// sync... etc
> 		RAMDirectory dir = new RAMDirectory(FSDirectory.getDirectory(directoryPath);
> 		IndexReader reader = IndexReader.open(dir, true);
> 		IndexSearcher searcher = new IndexSearcher(reader);
> 		searchers.put(index, searcher);
> 	}
> 	
> 	public static IndexSearcher get(IndexName index, ...) {
> 		// sync... etc
> 		return searchers.get(index);
> 	}
> 	
> 	// After an indexing service appends the existing FS Directory, this reloads it.
> 	public static void reload(IndexName index, ...) {
> 		// sync... etc
> 		IndexSearcher searcher = searchers.get(index);
> 		FSDirectory fsDirectory = FSDirectory.getDirectory(directoryPath);
> 		Directory ramDirectory = searcher.getIndexReader().directory();
> 		Directory.copy(fsDirectory, ramDirectory, false);
> 	}
>
> I've tries some theme and variations on this with the same issue.
>
> TIA!
>
> gd
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>   



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message