Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Subject: Reloading RAM Directory from updated FS Directory
Date: Tue, 9 Jun 2009 13:59:00 -0400
Message-ID: <6E43574E3326BB44BD95FDCF2C59827E01503982@HFEXNY1.HARRYFOX.COM>
Thread-Topic: Reloading RAM Directory from updated FS Directory
Thread-Index: AcnpK/eM4a6eLXK/Q7iTCUz9fL11RQ==
From: "Diamond, Greg" <GDiamond@harryfox.com>
To: <java-user@lucene.apache.org>

Hi All -

What is the best way to load a RAM Directory from a FS Directory, and =
periodically reload the RAM Directory to pick up new documents?

The scenario I have is I create several large directories which I create =
to a file system, then load them into ram for faster searching.
They takes several hours to create, so i want to retain a file copy so =
in the event of a service/server crash or reboot, they can load again
in a few seconds.  Once a day I append or recreate the FS Directories =
with new entries, and then reload the RAM Directories to pick up the
new entries.

Problem is, what I am doing seems to cause a memory leak.  The =
directories take ~12 GB of RAM to load, but bloats
to 24 GB (the -Xmx setting) after a few reloads and stays there.

	// Map to hold single instance of the IndexSearcher, 1 per RAM =
Directory, to be reused across requests.
	private static final Map<IndexName, IndexSearcher> searchers =3D new =
HashMap<IndexName, IndexSearcher>();
=09
	// Creates the IndexSearcher and stores it in the static map.
	public static void load(IndexName index, ...) {
		// sync... etc
		RAMDirectory dir =3D new =
RAMDirectory(FSDirectory.getDirectory(directoryPath);
		IndexReader reader =3D IndexReader.open(dir, true);
		IndexSearcher searcher =3D new IndexSearcher(reader);
		searchers.put(index, searcher);
	}
=09
	public static IndexSearcher get(IndexName index, ...) {
		// sync... etc
		return searchers.get(index);
	}
=09
	// After an indexing service appends the existing FS Directory, this =
reloads it.
	public static void reload(IndexName index, ...) {
		// sync... etc
		IndexSearcher searcher =3D searchers.get(index);
		FSDirectory fsDirectory =3D FSDirectory.getDirectory(directoryPath);
		Directory ramDirectory =3D searcher.getIndexReader().directory();
		Directory.copy(fsDirectory, ramDirectory, false);
	}

I've tries some theme and variations on this with the same issue.

TIA!

gd


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org