lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject Re: 2.1-dev memory leak?
Date Fri, 01 Dec 2006 05:16:11 GMT
Yeah, in this case, I'm running out of memory, and open file descriptors are, I think, just
an indicator that IndexSearchers are not getting closed properly.  I've already increased
the open file descriptors limit, but I'm limited to 2GB of RAM on a 32-bit box.

I'll try explicitly closing searchers now....some number of minutes later - bingo, that takes
care of the leak.

For the curious, this is what I think happened (the app in question is http://www.simpy.com/
).  The app was burning the CPU like crazy (imagine nearly double-digit load).  The IO didn't
seem crazy (vmstat/iostat didn't show much bi/bo).  Still, after some digging around the source
code, I found a place where I was just going through a few thousand Hits on each search against
one particular Lucene index.  I fixed that problem (nothing to do with IndexSearcher opening/closing),
and at the same time I upgraded to Lucene 2.1-dev.  This fix doubled the performance of the
whole service.  Instead of handling N requests/second, it was now handling 2-3*N requests/second.
 The CPU is still running pretty hot, but the service is handling more search traffic.  My
guess is that this increase in performance and throughput simply increased the number of different
Lucene indices (there are tens of thousands of them per box) being searches, and exposed an
issue that was either hidden before, or that the JVM was able to handle before, and now it
no longer could.  I wish I had a little better grip on what exactly happened here, but I've
got a stable application again.

Thanks for the help!

Otis

----- Original Message ----
From: Yonik Seeley <yonik@apache.org>
To: java-user@lucene.apache.org
Sent: Thursday, November 30, 2006 6:06:50 PM
Subject: Re: 2.1-dev memory leak?

On 11/30/06, Chris Hostetter <hossman_lucene@fucit.org> wrote:
> : IndexSearchers open.  The other ones I "let go" without an explicit
> : close() call.  The assumption is that the old IndexSearchers "expire",
> : that they get garbage collected, as I'm no longer holding references to
> : them.
>
> yeah ... that just seems really bad in general, i would try to explicitly
> close any searcher your purge from your cache.

Yeah, GC may kick off when memory gets low, but the GC system
unfortunately doesn't know anything about when file descriptors get
low.

-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org





---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message