lucenenet-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Oliver Donald" <oliver.don...@SnowValley.com>
Subject RE: Lucene memory usage
Date Tue, 31 Oct 2006 15:11:56 GMT
Close() method? Yes, that might have something to do with it ;)

I never considered that because we've been using Lucene without an issue
for quite a while, so I thought our wrapper was ok and it was something
I had done or a Lucene quirk, but I guess nothing has stressed it quite
as badly as my extensions...turns out our wrapper wasn't closing the
searcher :(

When is the best time to close the searcher though? If I close it after
I receive the Hits collection, I get a null reference exception from
FSDirectory.cs when I try and I call Hits.Doc(Int32 n).

This is for a webpage - is the best time to close the reader in the
Unload() event for the page?

Thanks,
Oli



-----Original Message-----
From: Murad James [mailto:murad@7northfield.com] 
Sent: 31 October 2006 14:14
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene memory usage

Yes, this sounds like something still has a reference to those string
objects.
 
If that doesn't work, post us a code snippet!

________________________________

From: Andy Berryman [mailto:andyb@channeladvisor.com]
Sent: Tue 31/10/2006 14:12
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene memory usage



Are you executing the "Close()" method on all your objects?  I didn't
investigate the fix in depth, but my understanding was that the fix was
to make another call to cleanup memory in the "Close()" method of the
Searcher/Reader object.

Andy

-----Original Message-----
From: Oliver Donald [mailto:oliver.donald@SnowValley.com]
Sent: Tuesday, October 31, 2006 9:04 AM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene memory usage

Unfortunately that's the version I am using :( Thanks for the reply
though!

I've got 1.6 million strings, and then 250,000 Term's and 250,000
TermInfo's. Tracing most of these back, they are referenced by
Hashtable's that appear in the allocation call stack at
IndexSearcher.Search() -> FieldSortedQueue.cctor(). So it looks like my
old search results are hanging around in memory and stopping the
associated resources being garbage collected...

Does this sound likely? Is there an obvious remedy or am I doing
something wrong? I'm using a single IndexSearcher, does this need to be
explicitly told to dispose of previous search results?

Thanks,
Oli


-----Original Message-----
From: Andy Berryman [mailto:andyb@channeladvisor.com]
Sent: 31 October 2006 12:40
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene memory usage

This sounds like a very similar problem that I was having with version
1.4.3.  A fix for this was made by George in version 1.9.1 build 4.

Andy

________________________________

From: Oliver Donald [mailto:oliver.donald@SnowValley.com]
Sent: Tue 10/31/2006 5:51 AM
To: lucene-net-user@incubator.apache.org
Subject: Lucene memory usage



Hi,

I'm relatively new to Lucene, but have been given a task of implementing
a complicated search. What I've ended up with is a page with search
results built from a number of queries (12 in total!) and as far as I
can tell there is no way to reduce this to less queries.

My problem is that Lucene ends up using up loads of RAM, but none of
this seems to get garbage collected. Each search adds another 10-30megs
of data, with occasional spikes of 50+ megs. This memory does not get
garbage collected, and a few searches down the line I get an out of
memory exception thrown.

So I ran a memory profiler on the app, and it turns out that 90%
(800,000+) of the memory is System.Strings, and many of these belong to
Lucene, often referenced by Term -> Term[] -> TermInfosReader ->
SegmentReader. What are these strings? Some kind of cache?

Is there anything I have to be careful of when performing groups of
searches at once? Is there any manual memory management I should be
doing? Is there anything I can do to reduce the amount of memory Lucene
uses?

My queries are generally simple, no more then 8 OR clauses each,
although they all have a range filter and sometimes a beginsWith filter
applied to them. I've been careful to avoid wildcards and ranges in the
main query itself!

Any help greatly appreciated!
Cheers,
Oli                   






Mime
View raw message