lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Diamond, Greg" <GDiam...@harryfox.com>
Subject Reloading RAM Directory from updated FS Directory
Date Tue, 09 Jun 2009 17:59:00 GMT
Hi All -

What is the best way to load a RAM Directory from a FS Directory, and periodically reload
the RAM Directory to pick up new documents?

The scenario I have is I create several large directories which I create to a file system,
then load them into ram for faster searching.
They takes several hours to create, so i want to retain a file copy so in the event of a service/server
crash or reboot, they can load again
in a few seconds.  Once a day I append or recreate the FS Directories with new entries, and
then reload the RAM Directories to pick up the
new entries.

Problem is, what I am doing seems to cause a memory leak.  The directories take ~12 GB of
RAM to load, but bloats
to 24 GB (the -Xmx setting) after a few reloads and stays there.

	// Map to hold single instance of the IndexSearcher, 1 per RAM Directory, to be reused across
requests.
	private static final Map<IndexName, IndexSearcher> searchers = new HashMap<IndexName,
IndexSearcher>();
	
	// Creates the IndexSearcher and stores it in the static map.
	public static void load(IndexName index, ...) {
		// sync... etc
		RAMDirectory dir = new RAMDirectory(FSDirectory.getDirectory(directoryPath);
		IndexReader reader = IndexReader.open(dir, true);
		IndexSearcher searcher = new IndexSearcher(reader);
		searchers.put(index, searcher);
	}
	
	public static IndexSearcher get(IndexName index, ...) {
		// sync... etc
		return searchers.get(index);
	}
	
	// After an indexing service appends the existing FS Directory, this reloads it.
	public static void reload(IndexName index, ...) {
		// sync... etc
		IndexSearcher searcher = searchers.get(index);
		FSDirectory fsDirectory = FSDirectory.getDirectory(directoryPath);
		Directory ramDirectory = searcher.getIndexReader().directory();
		Directory.copy(fsDirectory, ramDirectory, false);
	}

I've tries some theme and variations on this with the same issue.

TIA!

gd


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message