Mailing-List: contact lucene-net-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: lucene-net-user@incubator.apache.org
Received-SPF: neutral (herse.apache.org: local policy)
From: "George Aroush" <george@aroush.net>
To: <lucene-net-user@incubator.apache.org>
Subject: RE: Lucene memory usage
Date: Tue, 31 Oct 2006 23:51:44 -0500
Message-ID: <004b01c6fd71$6e6c9760$6401a8c0@aroushlt>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
Thread-Index: 
 Acb82oBpM95HLuwuSiClNiVx8TMcMgADyS/yAAFxP8AAAcKysAAAGIF7AABP22AAAf5MMAAANUtwAAIx5aoAARebIAAYZxAA
In-Reply-To: <403D5F429C79EE4199D99529BB153AD13C94E2@sv-mail.snowvalley.com>

Hi Oli,

If you can post some example code, it can go a long way for someone here to
see what's going on and offer some help.

I have written webserver application as well as ASP.NET application where
Lucene.Net keeps track of 3 indexes.  Memory usages has been steady.

>From this email thread, like Andy said, my guess is that you are not closing
a searcher, or opening a new searcher over and over.  Try this simple test,
make your searcher static and global.  Don't forgot to do the same to the
analyzer.

Regards,

-- George Aroush

-----Original Message-----
From: Oliver Donald [mailto:oliver.donald@SnowValley.com] 
Sent: Tuesday, October 31, 2006 12:11 PM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene memory usage

I've begun sharing a single instance and the memory usage is much
better...its still creeping up, but at a slightly less scary 0-5megs per
search.

Anyways thanks everyone for your help!
Oli

-----Original Message-----
From: Murad James [mailto:murad@7northfield.com]
Sent: 31 October 2006 16:29
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene memory usage

GC.SuppressFinalize stops .net garbage collection (GC) calling a ~finalizer
(a default method used to clean up the object) as the Close() method wants
GC to know that it has done everything that needs to be done.
 
It does seem like you have a common or garden problem with a reference
hanging around - this would show the behaviour you describe. By that, I mean
that there is something holding a reference to the search strings, perhaps
indirectly, that is preventing them being released. However, my knowledge of
dot lucene is not deep enough to be certain that it is not something within
dot lucene.

________________________________

From: Oliver Donald [mailto:oliver.donald@SnowValley.com]
Sent: Tue 31/10/2006 16:17
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene memory usage


Hi,

I've put the call in the page unload() event, as it seems safest. I realize
its more efficient to re-use the Searcher, but I want to isolate the leak
first.

Unfortunately memory usage is still going up at pretty much the same rate;
its quite jumpy, so its hard to tell, but I still get the out of memory
exception :(

Looking in the memory profiler, the references are the same as before I
added the Close() call on the IndexSearcher.

I noticed that in the Close method for the IndexSearcher, there are calls to
GC.SuppressFinalize(), what are they there for?

Thanks,
Oli


-----Original Message-----
From: Andy Berryman [mailto:andyb@channeladvisor.com]
Sent: 31 October 2006 15:25
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene memory usage

You'll most likely have to test different scenarios to determine the best
location for this call.  But you'll have to wait till after you have
gathered all the results that you want from the "Hits.Doc()" method call. 

One thing that you'll also want to take into account is that you get better
search performance for consequtive searches by keeping the Searcher object
alive and reusing it.  But in some cases this isnt realistic.  One thing to
take note of if you decide to do some sort of caching of the Searcher object
is that it is only aware of documents that were in the index at the time the
object was opened.

Andy

-----Original Message-----
From: Oliver Donald [mailto:oliver.donald@SnowValley.com]
Sent: Tuesday, October 31, 2006 10:12 AM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene memory usage

Close() method? Yes, that might have something to do with it ;)

I never considered that because we've been using Lucene without an issue for
quite a while, so I thought our wrapper was ok and it was something I had
done or a Lucene quirk, but I guess nothing has stressed it quite as badly
as my extensions...turns out our wrapper wasn't closing the searcher :(

When is the best time to close the searcher though? If I close it after I
receive the Hits collection, I get a null reference exception from
FSDirectory.cs when I try and I call Hits.Doc(Int32 n).

This is for a webpage - is the best time to close the reader in the
Unload() event for the page?

Thanks,
Oli


-----Original Message-----
From: Murad James [mailto:murad@7northfield.com]
Sent: 31 October 2006 14:14
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene memory usage

Yes, this sounds like something still has a reference to those string
objects.

If that doesn't work, post us a code snippet!

________________________________

From: Andy Berryman [mailto:andyb@channeladvisor.com]
Sent: Tue 31/10/2006 14:12
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene memory usage


Are you executing the "Close()" method on all your objects?  I didn't
investigate the fix in depth, but my understanding was that the fix was to
make another call to cleanup memory in the "Close()" method of the
Searcher/Reader object.

Andy

-----Original Message-----
From: Oliver Donald [mailto:oliver.donald@SnowValley.com]
Sent: Tuesday, October 31, 2006 9:04 AM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene memory usage

Unfortunately that's the version I am using :( Thanks for the reply though!

I've got 1.6 million strings, and then 250,000 Term's and 250,000
TermInfo's. Tracing most of these back, they are referenced by Hashtable's
that appear in the allocation call stack at
IndexSearcher.Search() -> FieldSortedQueue.cctor(). So it looks like my old
search results are hanging around in memory and stopping the associated
resources being garbage collected...

Does this sound likely? Is there an obvious remedy or am I doing something
wrong? I'm using a single IndexSearcher, does this need to be explicitly
told to dispose of previous search results?

Thanks,
Oli


-----Original Message-----
From: Andy Berryman [mailto:andyb@channeladvisor.com]
Sent: 31 October 2006 12:40
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene memory usage

This sounds like a very similar problem that I was having with version
1.4.3.  A fix for this was made by George in version 1.9.1 build 4.

Andy

________________________________

From: Oliver Donald [mailto:oliver.donald@SnowValley.com]
Sent: Tue 10/31/2006 5:51 AM
To: lucene-net-user@incubator.apache.org
Subject: Lucene memory usage


Hi,

I'm relatively new to Lucene, but have been given a task of implementing a
complicated search. What I've ended up with is a page with search results
built from a number of queries (12 in total!) and as far as I can tell there
is no way to reduce this to less queries.

My problem is that Lucene ends up using up loads of RAM, but none of this
seems to get garbage collected. Each search adds another 10-30megs of data,
with occasional spikes of 50+ megs. This memory does not get garbage
collected, and a few searches down the line I get an out of memory exception
thrown.

So I ran a memory profiler on the app, and it turns out that 90%
(800,000+) of the memory is System.Strings, and many of these belong to
Lucene, often referenced by Term -> Term[] -> TermInfosReader ->
SegmentReader. What are these strings? Some kind of cache?

Is there anything I have to be careful of when performing groups of searches
at once? Is there any manual memory management I should be doing? Is there
anything I can do to reduce the amount of memory Lucene uses?

My queries are generally simple, no more then 8 OR clauses each, although
they all have a range filter and sometimes a beginsWith filter applied to
them. I've been careful to avoid wildcards and ranges in the main query
itself!

Any help greatly appreciated!
Cheers,
Oli