subversion-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marc Strapetz <marc.strap...@syntevo.com>
Subject Re: 1.9 JavaHL memory leak in ISVNRemote#status
Date Tue, 28 Apr 2015 16:03:27 GMT
Hi Brane,

On 28.04.2015 07:36, Branko Čibej wrote:
> On 24.04.2015 14:11, Branko Čibej wrote:
>>
>> Hi Marc,
>>
>> Just a quick note: your last msg jogged my memory and I think I know
>> the root cause of the leak: improper JNI frame management within a
>> loop. If I'm right, I can both fix the leak and remove the
>> close-stream requirement I just added.
>>
>> On 24 Apr 2015 11:00 am, "Marc Strapetz" <marc.strapetz@syntevo.com
>> <mailto:marc.strapetz@syntevo.com>> wrote:
>>
>>     On 24.04.2015 06 <tel:24.04.2015%2006>:34, Branko Čibej wrote:
>>
>>         On 22.03.2015 05 <tel:22.03.2015%2005>:06, Branko Čibej wrote:
>>
>>             On 21.03.2015 16 <tel:21.03.2015%2016>:23, Branko Čibej wrote:
>>
>>                 On 19.03.2015 11:43, Marc Strapetz wrote:
>>
>>                     Attached example performs an endless series of
>>                     remote status against
>>                     the Subversion repository. When invoked with
>>                     -Xmx24M, the VM will run
>>                     out of memory soon. Monitoring with jvisualvm
>>                     shows that the used heap
>>                     size constantly grows. Monitoring with the Task
>>                     Manager shows that the
>>                     allocated memory grows even more (significantly).
>>                     Looks like a memory
>>                     leak, for which a large amount of native memory is
>>                     involved, too.
>>
>>                     Tested on Windows 8.1 with almost latest
>>                     Subversion 1.9 JavaHL builds.
>>
>>                 I can confirm that this happens on the Mac, too, and
>>                 it's not a garbage
>>                 collector artefact. I'm trying to trace where the leak
>>                 is happening ...
>>                 valgrind with APR pool debugging doesn't tell me much
>>                 (no surprise there).
>>
>>             Just to make sure we weren't doing something bad in our
>>             libraries, I
>>             wrote a small C program that does the same as your Java
>>             example (Ev2
>>             shims included), and memory usage is completely steady. So
>>             it is
>>             something in JavaHL, but I have no clue yet what the
>>             problem is.
>>
>>
>>         I have to say this was one of the more "interesting" bug-hunts
>>         in my not
>>         entirely boring career, and that's not really obvious from the fix
>>         itself. :)
>>
>>         http://svn.apache.org/r1675771
>>
>>         Marc: this will not be in RC1, but please give the patch a
>>         spin and let
>>         me know if it fixes your problem. I tested this with the Java
>>         program
>>         you attached to your original report, and heap size no longer
>>         grows
>>         without bounds.
>>
>>
>>     Great hunt, Brane! The native leak seems to be fixed. I've run my
>>     remote status loop with -Xmx24M and still get an OOME after ~170
>>     loop iterations. The memory leak is significantly smaller and this
>>     time it seems to be in the Java part. According to the profiler,
>>     most memory is allocated by HashMap and friends, referenced from
>>     JNI code. Only two org.apache.subversion classes show up, but I
>>     guess they indicate the source of the leak:
>>
>>     org.apache.subversion.javahl.types.Checksum (~10K instances)
>>     org.apache.subversion.javahl.types.NativeInputStream (~10K instances)
>>
>>     Let me know, if you more profiler statistics will be helpful.
>>
>
> So I've been looking at this in depth. At first I thought that one of
> the problems was that we didn't release JNI local references; I added
> code to make sure this happens in the status callbacks (not committed
> yet) and I verified that all the native wrapped objects do get
> finalized. However, the Java objects still hang around.
>
> One of the problems is that all the callbacks happen within the scope of
> the ISVNReporter.finishReport call, which means that the whole edit
> drive is considered a single JNI call (despite the callbacks to Java)
> and the garbage collector can't reclaim space for the objects created
> within JNI during that time. But even a forced GC after the report is
> done and the remote session disposed won't release all the native
> references. I'm a bit stumped here ... JVM's built-in memory profiler
> shows the live references and where they're allocated, but doesn't show
> why they're not released even when I explicitly create and destroy JNI
> frames.

Can you please commit your current state somewhere or send me a patch? I 
can give this one a try in JProfiler and see whether I can gather some 
more useful information.

-Marc

Mime
View raw message