nutch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kirby Bohling <kirby.bohl...@gmail.com>
Subject Re: Possible memory leak in Nutch-1.0 ?
Date Thu, 10 Sep 2009 15:22:40 GMT
On Thu, Aug 20, 2009 at 10:42 AM, Mark Round <mark.round@ahc.uk.com> wrote:
>> not sure but maybe it is possible that the PluginRepository has the
>> memory leak. i think the cache (the weakhashmap) is growing and
> growing.
>
> Is this the same issue as reported here :
> https://issues.apache.org/jira/browse/NUTCH-356 ?
>
> It may be adding to my troubles, but I suspect my immediate problem is
> more likely to be directly related to the hundreds/thousands of threads
> being created. Do you see that problem with your test code ?
>

Mark,

   You reminded me off-list that I forgot to follow up on this.
Thanks for the reminder, busy lately.  Replying on-list so others can
see.

   It's not that the threads are hung.  They'd all exit eventually.
It's just they aren't existing fast enough, so you are building up a
large number of them.

It appears the problem is trivial to work around.  Actually fixing it
will be harder.  I'm not using the stock Nutch, but I hit the
customized one I run with similar patches to what you describe.  The
crux of the problem is that InterruptedException's appear to be
swallowed (that's a guess with no debugging), and the default block
time inside of "FetchedSegments.SegmentUpdater.run()" is too long.

It's a simple enqueue/dequeue rate problem.  If the interruption is
swallowed by client code, the thread will block for 60 seconds prior
to exiting.  So you create one every time through the main loop, but
it potentially has to wait 60 seconds before it dies.  On my machine,
changing that from 60000 to 1000 "fixes" the problem.  If the search
ran faster, this would still be a problem, 1000 just made the dequeue
at least as fast as enqueue.

   As a temporary fix, I'd make that change and see if it works for
you.  The real solution is to find the culprit that is eating
InterruptedException's and ensure that they keep resetting the
interruption until they get to the top level.


I'm assuming that the overhead for that loop isn't that bad.  So just
sleeping for 1 second won't "hurt".  However, that might be a
significant waste of CPU time (if it takes 4 seconds to process that
loop, it'd be 80% busy loop).  I don't think it is, but I haven't
actually validated that assumption.

   The other obvious hack is to do something like:

if (!stopRequested) {
    Thread.sleep(60000); // Pick 1000 if that makes you happier.
}


   Hope that helps, if it does, great.  If it doesn't say something.
Potentially a change that should be integrated into Nutch proper.
Either way, I probably should file a JIRA issue about the interruption
exceptions not being properly handled.

   I got it to run for ~10-15 minutes, before I'm hitting a GC
overhead issues while running inside of Eclipse and a debugger.  I'm
not sure about that.  If you have that problem, I can try and look
into that also.  Figured this might get you over the hump that we can
both work on the next issue if need be.

         Kirby

Mime
View raw message