river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Thompson <br...@systap.com>
Subject RE: DGC threads issue
Date Fri, 13 Jan 2012 12:28:32 GMT
Peter,

The DGC Threads are in the server JVM.  They are nearly all for the exported Futures.

Thanks,
Bryan

> -----Original Message-----
> From: Peter Jones [mailto:pcj@roundroom.net] 
> Sent: Friday, January 13, 2012 12:04 AM
> To: user@river.apache.org
> Cc: dev@river.apache.org
> Subject: Re: DGC threads issue
> 
> Bryan,
> 
> DGC threads should not be per exported object.  Generally 
> speaking, they tend to be per endpoint (at which there are 
> one or more remote objects exported).  Are you using any sort 
> of custom endpoint implementation?  Problems like this can 
> occur when an endpoint implementation doesn't implement 
> Object.equals and hashCode appropriately, so the expected 
> batching of threads (and communication) per endpoint does not occur.
> 
> It might help to see, from a thread dump, exactly which DGC 
> threads are causing this problem.  And they are in the server 
> JVM (with all the exported remote objects), not the remote 
> callers' JVM(s)?
> 
> -- Peter
> 
> 
> On Jan 12, 2012, at 3:45 PM, Tom Hobbs wrote:
> 
> > Hi Bryan,
> > 
> > Sorry that no one got back to you about this.  I'm afraid 
> that I don't 
> > know the answer to your question, I've copied the dev list 
> into this 
> > email in case someone who monitors that list (but not this one) has 
> > any ideas.
> > 
> > Best regards,
> > 
> > Tom
> > 
> > On Thu, Jan 12, 2012 at 2:29 PM, Bryan Thompson 
> <bryan@systap.com> wrote:
> >> Just to follow up on this thread myself.  I modified the 
> pattern to return a "thick" future rather than a proxy for 
> the future.  This caused the RMI call to wait on the server 
> until the future was done and then sent back the outcome.  
> This "fixed" the DGC memory/thread leak by reducing the 
> number of exported proxies drammatically.
> >> 
> >> In terms of best practices, is distributed DGC simply not 
> useful for exported objects with short life spans?  Can it 
> only be used with proxies for relatively long lived services?
> >> 
> >> Thanks,
> >> Bryan
> >> 
> >>> -----Original Message-----
> >>> From: Bryan Thompson
> >>> Sent: Tuesday, January 03, 2012 12:06 PM
> >>> To: user@river.apache.org
> >>> Subject: DGC threads issue
> >>> 
> >>> Hello,
> >>> 
> >>> Background:
> >>> 
> >>> I am seeing what would appear to be one DGC thread allocated per 
> >>> exported object.  This is using River 2.2 and Sun JDK 1.6.0_17.  
> >>> Relevant configuration parameters are below.
> >>> 
> >>> I am observing problems with the DGC threads not being 
> retired on a 
> >>> timely basis.  The exported objects are proxies for Futures which 
> >>> are being executed on the service.  The code pattern is such that 
> >>> the proxied Future goes out of lexical scope quite 
> quickly.  E.g., 
> >>> rmiCallReturningProxyForFuture().get().
> >>> 
> >>> Under a modest load, a large number of such Futures are exported 
> >>> which results in a large number of long lived DGC threads.  This 
> >>> turns into a problem for the JVM due to the stack allocation per 
> >>> thread.  Presumably this is not good for other reasons as well 
> >>> (e.g., scheduling).
> >>> 
> >>> I have tried to override the leaseValue and checkInterval 
> defaults 
> >>> per the configuration options below.  I suspect that the lease 
> >>> interval is somehow not being obeyed, which is presumably 
> a problem 
> >>> on my end.  However, I can verify that the configuration 
> values are 
> >>> in fact showing up in
> >>> System.getProperties() for at least some of the JVMs 
> involved (the 
> >>> one which drives the workload and the one that I am 
> monitoring with 
> >>> the large number of DGC lease threads).
> >>> 
> >>> Some questions:
> >>> 
> >>> Is this one-thread-per-exported proxy the expected 
> behavior when DGC 
> >>> is requested for the exported object?
> >>> 
> >>> The DGC lease checker threads appear to expire ~14 - 15 minutes 
> >>> after I terminate the process which was originating the RMI 
> >>> requests.  This is close the sum of the default 
> leaseValue (10m) and 
> >>> checkInterval (5m) parameters, but maybe there is some 
> other timeout 
> >>> which is controlling this?  If this is the sum of those 
> parameters, 
> >>> why would the DGC lease threads live until the sum of 
> those values?  
> >>> I thought that the lease would expire after the leaseValue (10m 
> >>> default).
> >>> 
> >>> Can the issue I am observing be caused by a low heap 
> pressure on the 
> >>> JVM to which the RMI proxies were exported?  If it fails 
> to GC those 
> >>> proxies, even though they are reachable, could that cause DGC to 
> >>> continue to retain those proxies on the JVM which exported them?
> >>> 
> >>> Is there any way to configure DGC to use a thread pool or to have 
> >>> the leases managed by a single thread?
> >>> 
> >>> Is it possible that there is an interaction with the 
> useNIO option?
> >>> 
> >>> Relevant options that I am using include:
> >>> 
> >>>    -Dcom.sun.jini.jeri.tcp.useNIO=true
> >>>    -Djava.rmi.dgc.leaseValue=30000
> >>>    -Dsun.rmi.dgc.checkInterval=15000
> >>>    -Dsun.rmi.transport.tcp.connectionPool=true
> >>> 
> >>> Thanks in advance,
> >>> Bryan
> 
> 
Mime
View raw message