river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Jones <...@roundroom.net>
Subject Re: DGC threads issue
Date Fri, 13 Jan 2012 14:31:11 GMT
Bryan,

I meant that it might help for the list to "see" the specific threads in question, as they
appear in a JVM thread dump (name, stack frames, etc.), just to be sure that we're talking
about the same thing.  There is more than one kind of thread related to DGC, and it seems
that the implementation has changed recently.  But I gather that Peter F. may have identified
the root cause.

Cheers,

-- Peter


On Jan 13, 2012, at 7:28 AM, Bryan Thompson wrote:

> Peter,
> 
> The DGC Threads are in the server JVM.  They are nearly all for the exported Futures.
> 
> Thanks,
> Bryan
> 
>> -----Original Message-----
>> From: Peter Jones [mailto:pcj@roundroom.net] 
>> Sent: Friday, January 13, 2012 12:04 AM
>> To: user@river.apache.org
>> Cc: dev@river.apache.org
>> Subject: Re: DGC threads issue
>> 
>> Bryan,
>> 
>> DGC threads should not be per exported object.  Generally 
>> speaking, they tend to be per endpoint (at which there are 
>> one or more remote objects exported).  Are you using any sort 
>> of custom endpoint implementation?  Problems like this can 
>> occur when an endpoint implementation doesn't implement 
>> Object.equals and hashCode appropriately, so the expected 
>> batching of threads (and communication) per endpoint does not occur.
>> 
>> It might help to see, from a thread dump, exactly which DGC 
>> threads are causing this problem.  And they are in the server 
>> JVM (with all the exported remote objects), not the remote 
>> callers' JVM(s)?
>> 
>> -- Peter
>> 
>> 
>> On Jan 12, 2012, at 3:45 PM, Tom Hobbs wrote:
>> 
>>> Hi Bryan,
>>> 
>>> Sorry that no one got back to you about this.  I'm afraid 
>> that I don't 
>>> know the answer to your question, I've copied the dev list 
>> into this 
>>> email in case someone who monitors that list (but not this one) has 
>>> any ideas.
>>> 
>>> Best regards,
>>> 
>>> Tom
>>> 
>>> On Thu, Jan 12, 2012 at 2:29 PM, Bryan Thompson 
>> <bryan@systap.com> wrote:
>>>> Just to follow up on this thread myself.  I modified the 
>> pattern to return a "thick" future rather than a proxy for 
>> the future.  This caused the RMI call to wait on the server 
>> until the future was done and then sent back the outcome.  
>> This "fixed" the DGC memory/thread leak by reducing the 
>> number of exported proxies drammatically.
>>>> 
>>>> In terms of best practices, is distributed DGC simply not 
>> useful for exported objects with short life spans?  Can it 
>> only be used with proxies for relatively long lived services?
>>>> 
>>>> Thanks,
>>>> Bryan
>>>> 
>>>>> -----Original Message-----
>>>>> From: Bryan Thompson
>>>>> Sent: Tuesday, January 03, 2012 12:06 PM
>>>>> To: user@river.apache.org
>>>>> Subject: DGC threads issue
>>>>> 
>>>>> Hello,
>>>>> 
>>>>> Background:
>>>>> 
>>>>> I am seeing what would appear to be one DGC thread allocated per 
>>>>> exported object.  This is using River 2.2 and Sun JDK 1.6.0_17.  
>>>>> Relevant configuration parameters are below.
>>>>> 
>>>>> I am observing problems with the DGC threads not being 
>> retired on a 
>>>>> timely basis.  The exported objects are proxies for Futures which 
>>>>> are being executed on the service.  The code pattern is such that 
>>>>> the proxied Future goes out of lexical scope quite 
>> quickly.  E.g., 
>>>>> rmiCallReturningProxyForFuture().get().
>>>>> 
>>>>> Under a modest load, a large number of such Futures are exported 
>>>>> which results in a large number of long lived DGC threads.  This 
>>>>> turns into a problem for the JVM due to the stack allocation per 
>>>>> thread.  Presumably this is not good for other reasons as well 
>>>>> (e.g., scheduling).
>>>>> 
>>>>> I have tried to override the leaseValue and checkInterval 
>> defaults 
>>>>> per the configuration options below.  I suspect that the lease 
>>>>> interval is somehow not being obeyed, which is presumably 
>> a problem 
>>>>> on my end.  However, I can verify that the configuration 
>> values are 
>>>>> in fact showing up in
>>>>> System.getProperties() for at least some of the JVMs 
>> involved (the 
>>>>> one which drives the workload and the one that I am 
>> monitoring with 
>>>>> the large number of DGC lease threads).
>>>>> 
>>>>> Some questions:
>>>>> 
>>>>> Is this one-thread-per-exported proxy the expected 
>> behavior when DGC 
>>>>> is requested for the exported object?
>>>>> 
>>>>> The DGC lease checker threads appear to expire ~14 - 15 minutes 
>>>>> after I terminate the process which was originating the RMI 
>>>>> requests.  This is close the sum of the default 
>> leaseValue (10m) and 
>>>>> checkInterval (5m) parameters, but maybe there is some 
>> other timeout 
>>>>> which is controlling this?  If this is the sum of those 
>> parameters, 
>>>>> why would the DGC lease threads live until the sum of 
>> those values?  
>>>>> I thought that the lease would expire after the leaseValue (10m 
>>>>> default).
>>>>> 
>>>>> Can the issue I am observing be caused by a low heap 
>> pressure on the 
>>>>> JVM to which the RMI proxies were exported?  If it fails 
>> to GC those 
>>>>> proxies, even though they are reachable, could that cause DGC to 
>>>>> continue to retain those proxies on the JVM which exported them?
>>>>> 
>>>>> Is there any way to configure DGC to use a thread pool or to have 
>>>>> the leases managed by a single thread?
>>>>> 
>>>>> Is it possible that there is an interaction with the 
>> useNIO option?
>>>>> 
>>>>> Relevant options that I am using include:
>>>>> 
>>>>>   -Dcom.sun.jini.jeri.tcp.useNIO=true
>>>>>   -Djava.rmi.dgc.leaseValue=30000
>>>>>   -Dsun.rmi.dgc.checkInterval=15000
>>>>>   -Dsun.rmi.transport.tcp.connectionPool=true
>>>>> 
>>>>> Thanks in advance,
>>>>> Bryan
>> 


Mime
View raw message