lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: 2.1-dev memory leak?
Date Thu, 30 Nov 2006 23:32:04 GMT
Otis Gospodnetic wrote:

> Wow, that was fast - java-user support is just as fast as I heard! ;)

Well let's withhold judgment until we see if that tool really works
correctly :)

> I'll try your patch shortly. Like I said, the bug may be in my application.  Here is
a clue.  Memory usage increases with the number of open files (file descriptors) on the system,
and lsof gives:
> 
> COMMAND     PID    USER   FD      TYPE     DEVICE      SIZE       NODE NAME
> ...
> java      24476   xxx *562r      REG      253,0    657314    8470761 /xxx/users/1/1-home/link/index/_20k.cfs
> java      24476   xxx *894r      REG      253,0    657314    8470761 /xxx/users/1/1-home/link/index/_20k.cfs
> java      24476   xxx *078r      REG      253,0    657314    8470761 /xxx/users/1/1-home/link/index/_20k.cfs
> java      24476   xxx *648r      REG      253,0    657314    8470761 /xxx/users/1/1-home/link/index/_20k.cfs
> ...
> 
> If I'm reading this right, this tells me that this same file has been opened a number
of different times (note the FD column, all file descriptors are different).  This must correspond
to multiple new IndexSearcher(...) calls, no?  Multiple new IndexSearcher(...) calls on the
same index are okay in my case - because the system has tens of thousands of separate indices,
it can't keep all IndexSearchers open at all times, so I use the LRU algo to keep only recently
used IndexSearchers open.  The other ones I "let go" without an explicit close() call.  The
assumption is that the old IndexSearchers "expire", that they get garbage collected, as I'm
no longer holding references to them.
> Iff I understand this correctly, the fact that I see these open file descriptors all
pointing to the same index file tells me that the old IndexSearchers are just hanging around
and are not getting cleaned up.
> 
> I can also see the number of file descriptors increasing with time:
> 
> $ /usr/sbin/lsof | grep -c '/1-home/link/index/_20k.cfs'
> 14
> $ /usr/sbin/lsof | grep -c '/1-home/link/index/_20k.cfs'
> 23
> 
> This may still not point to my app having the bug, but it points to something not releasing
IndexSearcher/IndexReader, as that is not getting GCed as before.  I did not change my logic
for creating new IndexSearchers (inlined in my previous email).  On the other hand, this app
has recently started getting a lot more search action, so perhaps it's just that the GC is
not cleaning things up fast enough....
> I happen to have an lsof output from the same system from July.  I see the same thing
there - a number of FDs open and pointing to the same .cfs index file.  Perhaps it's just
that the JVM GC was able to clean things up then, and now it can't, because the CPU is maxed
out.... really maxed out.

Otis, one more thing you could try is to call
SegmentInfos.setInfoStream(System.out).  This will cause lockless to
print details whenever it has to retry loading a segments_N file (just
in case something is going horribly wrong...).  This should only happen
when there is contention (writer committing just as reader is opening)
which depending on your application may or may not ever happen.

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message