river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Thompson <br...@systap.com>
Subject RE: RE: [Fwd: Re: DGC threads issue]
Date Fri, 13 Jan 2012 12:53:50 GMT
Tom,

Ok.  I will fetch the code for 2.2 and work through the build process.  I have filed an issue
for this [1].  If someone can post a patch for that issue against 2.2, I will build the modified
JARs and verify whether the problem is resolved by that change.

Thanks,
Bryan

PS: River 2.2 is listed as an "Unreleased Version" in Jira.

[1] https://issues.apache.org/jira/browse/RIVER-403

> -----Original Message-----
> From: Tom Hobbs [mailto:tvhobbs@googlemail.com] 
> Sent: Friday, January 13, 2012 7:35 AM
> To: user@river.apache.org
> Subject: Re: RE: [Fwd: Re: DGC threads issue]
> 
> The easiest way would be to the source distribution of River. 
>  Unpack it somewhere.  Modify the code and then run the ant 
> task called "build-all" (I think, whatever it's called it 
> default target for the build.xml).  The build file should be 
> in the root of the unpacked source.
> 
> I don't think Peter created a jira fir it because I've not 
> seen a notification email.  I'm also unsure what code libel 
> he's referring to.
> 
> Sent via mobile device, please forgive typos and spacing errors.
> 
> On 13 Jan 2012 12:30, "Bryan Thompson" <bryan@systap.com> wrote:
> 
> > Peter,
> >
> > I would be happy to test out a fix on this.  Can you point out the 
> > modified line and link me to any directions on how to rebuild the 
> > necessary jar(s)?
> >
> > Did you already file an issue for this?  If not, I will 
> file an issue 
> > and then update it when I test out this fix.
> >
> > Thanks,
> > Bryan
> >
> > > -----Original Message-----
> > > From: Peter Firmstone [mailto:jini@zeus.net.au]
> > > Sent: Friday, January 13, 2012 3:21 AM
> > > To: dev@river.apache.org
> > > Cc: Bryan Thompson; user@river.apache.org
> > > Subject: Re: [Fwd: Re: DGC threads issue]
> > >
> > > Ok, found the problem, when I fixed River-142, I introduced a new 
> > > bug, after creating a new DGC thread, I didn't set the 
> running flag 
> > > to true, so each time, a new thread is created.
> > >
> > > Thankfully it's a very easy fix.
> > >
> > > Cheers,
> > >
> > > Peter.
> > >
> > > Peter Jones wrote:
> > > > Peter,
> > > >
> > > > That internal Executor interface permits neither
> > > synchronous execution in the calling thread ("this method itself 
> > > must not block") nor any small bound on the number of 
> Runnables that 
> > > can be executed concurrently ("execution of a given 
> action must not 
> > > be delayed indefinitely in order to complete execution of a 
> > > different action passed to a different invocation of this 
> method").  
> > > In other words, concurrency-wise, it should be equivalent to "new 
> > > Thread(runnable, name)", or what the result of
> > > j.u.c.Executors.newCachedThreadPool() would do.  
> Otherwise, callers 
> > > would not function correctly, because they expect to be 
> able to pass 
> > > Runnables that execute indefinitely, not just short-lived work 
> > > tasks.
> > > >
> > > > The real purpose of this internal Executor/ThreadPool was
> > > to provide an internal alternative to "new Thread(...)" 
> that takes 
> > > care of certain things in common, like:
> > > >
> > > > - thread reuse instead of creation when reasonable 
> (idle timeout)
> > > > - but still with threads (re)named for the task being executed
> > > > - insulation of any thread creation permission requirement from 
> > > > context calling execute (NewThreadAction)
> > > > - but instead requiring direct user of thread pool to be
> > > trusted not
> > > > to abuse that insulation, with permission requirement 
> for access 
> > > > (ThreadPoolPermission and GetThreadPoolAction)
> > > > - common logging of uncaught exceptions
> > > >
> > > > -- Peter
> > > >
> > > >
> > > > On Jan 12, 2012, at 7:53 PM, Peter Firmstone wrote:
> > > >
> > > >
> > > >> ThreadPool implements the com.sun.jini.thread.Executor 
> interface.
> > > >>
> > > >> Because the interface states it should not block, I think
> > > the calling thread should execute the task when the thread pool 
> > > becomes saturated, rather than continue to create new 
> threads as per 
> > > the current implementation.  This will ensure that the task is 
> > > completed, the calling thread, if it uses a sequence of 
> tasks with 
> > > dependencies will have to ensure that it submits the 
> tasks in order.  
> > > Since only the caller knows the order, it makes sense for 
> this to be 
> > > the callers responsibility.  As a result the executor 
> will honor the 
> > > non blocking contract.  For that reason we'll use a zero length 
> > > queue, probably a SynchronousQueue.
> > > >>
> > > >> Regards,
> > > >>
> > > >> Peter.
> > > >>
> > > >> /**
> > > >> * Executor is an abstraction for a thread factory or
> > > thread pool for
> > > >> * executing actions asynchronously.
> > > >> *
> > > >> * @author    Sun Microsystems, Inc.
> > > >> *
> > > >> */
> > > >> public interface Executor {
> > > >>
> > > >>   /**
> > > >>    * Executes the given Runnable action asynchronously in
> > > some thread.
> > > >>    *
> > > >>    * The implementation may create a new thread to execute
> > > the action,
> > > >>    * or it may execute the action in an existing thread.
> > > >>    *
> > > >>    * The execution of a given action must not be delayed
> > > indefinitely
> > > >>    * in order to complete execution of a different action
> > > passed to a
> > > >>    * different invocation of this method.  In other words, the
> > > >>    * implementation must assume that there may be
> > > arbitrary dependencies
> > > >>    * between actions passed to this method, so it needs to
> > > be careful
> > > >>    * to avoid potential deadlock by delaying execution of
> > > one action
> > > >>    * indefinitely until another completes.
> > > >>    *
> > > >>    * Also, this method itself must not block, because it
> > > may be invoked
> > > >>    * by code that is serially processing data to produce
> > > multiple such
> > > >>    * arbitrarily-dependent actions that need to be executed.
> > > >>    *
> > > >>    * @param    runnable the Runnable action to execute
> > > >>    *
> > > >>    * @param    name string to include in the name of the
> > > thread used
> > > >>    * to execute the action
> > > >>    */
> > > >>   void execute(Runnable runnable, String name); }
> > > >>
> > > >> Peter Firmstone wrote:
> > > >>
> > > >>> Thanks Brian,
> > > >>>
> > > >>> Looking at our implementation code, DGC uses an Executor
> > > called ThreadPool, it's javadoc states:
> > > >>>
> > > >>> /**
> > > >>> * ThreadPool is a simple thread pool implementation of
> > > the Executor
> > > >>> * interface.
> > > >>> *
> > > >>> * A new task is always given to an idle thread, if one is
> > > available;
> > > >>> * otherwise, a new thread is always created.  There is no 
> > > >>> minimum
> > > >>> * warm thread count, nor is there a maximum thread count
> > > (tasks are
> > > >>> * never queued unless there are sufficient idle threads to 
> > > >>> execute
> > > >>> * them).
> > > >>> *
> > > >>> * New threads are created as daemon threads in the thread
> > > group that
> > > >>> * was passed to the ThreadPool instance's constructor.  Each 
> > > >>> thread's
> > > >>> * name is the prefix NewThreadAction.NAME_PREFIX 
> followed by the 
> > > >>> name
> > > >>> * of the task it is currently executing, or "Idle" if it is 
> > > >>> currently
> > > >>> * idle.
> > > >>>
> > > >>> ThreadPool predates Java 5, it looks like we can fix this
> > > by using an Executor from Java 5, we can look at limiting 
> the number 
> > > of threads created based on available CPU's and a scaling 
> factor and 
> > > place the tasks in a BlockingQueue, so if the queue is filled, it 
> > > blocks.
> > > >>>
> > > >>> Can you report the issue as a Bug on Jira for me, I'll
> > > fix this before the next release.
> > > >>>
> > > >>> Regards,
> > > >>>
> > > >>> Peter.
> > > >>>
> > > >>> Peter Firmstone wrote:
> > > >>>
> > > >>>> Hi Peter,
> > > >>>>
> > > >>>> I was wondering if you had any thoughts on this post
> > > from Bryan on River users?
> > > >>>>
> > > >>>> Hope you don't mind me asking ;)
> > > >>>>
> > > >>>> Best Regards,
> > > >>>>
> > > >>>> Peter Firmstone.
> > > >>>>
> > > >>>>
> > > 
> -------------------------------------------------------------------
> > > >>>> -----
> > > >>>>
> > > >>>> Subject:
> > > >>>> Re: DGC threads issue
> > > >>>> From:
> > > >>>> Tom Hobbs <tvhobbs@googlemail.com>
> > > >>>> Date:
> > > >>>> Thu, 12 Jan 2012 20:45:01 +0000
> > > >>>> To:
> > > >>>> user@river.apache.org, dev@river.apache.org
> > > >>>>
> > > >>>> To:
> > > >>>> user@river.apache.org, dev@river.apache.org
> > > >>>>
> > > >>>>
> > > >>>> Hi Bryan,
> > > >>>>
> > > >>>> Sorry that no one got back to you about this.  I'm 
> afraid that 
> > > >>>> I don't know the answer to your question, I've 
> copied the dev 
> > > >>>> list into this email in case someone who monitors that list

> > > >>>> (but not this one) has any ideas.
> > > >>>>
> > > >>>> Best regards,
> > > >>>>
> > > >>>> Tom
> > > >>>>
> > > >>>> On Thu, Jan 12, 2012 at 2:29 PM, Bryan Thompson
> > > <bryan@systap.com> wrote:
> > > >>>>
> > > >>>>
> > > >>>>> Just to follow up on this thread myself.  I modified
> > > the pattern to return a "thick" future rather than a 
> proxy for the 
> > > future.  This caused the RMI call to wait on the server until the 
> > > future was done and then sent back the outcome.  This "fixed" the 
> > > DGC memory/thread leak by reducing the number of exported proxies 
> > > drammatically.
> > > >>>>>
> > > >>>>> In terms of best practices, is distributed DGC simply
> > > not useful for exported objects with short life spans?  
> Can it only 
> > > be used with proxies for relatively long lived services?
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Bryan
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>>> -----Original Message-----
> > > >>>>>> From: Bryan Thompson
> > > >>>>>> Sent: Tuesday, January 03, 2012 12:06 PM
> > > >>>>>> To: user@river.apache.org
> > > >>>>>> Subject: DGC threads issue
> > > >>>>>>
> > > >>>>>> Hello,
> > > >>>>>>
> > > >>>>>> Background:
> > > >>>>>>
> > > >>>>>> I am seeing what would appear to be one DGC thread
> > > allocated per
> > > >>>>>> exported object.  This is using River 2.2 and Sun
JDK
> > > 1.6.0_17.
> > > >>>>>> Relevant configuration parameters are below.
> > > >>>>>>
> > > >>>>>> I am observing problems with the DGC threads not being
> > > retired on
> > > >>>>>> a timely basis.  The exported objects are proxies

> for Futures 
> > > >>>>>> which are being executed on the service.  The code

> pattern is 
> > > >>>>>> such that the proxied Future goes out of lexical 
> scope quite 
> > > >>>>>> quickly.  E.g., rmiCallReturningProxyForFuture().get().
> > > >>>>>>
> > > >>>>>> Under a modest load, a large number of such Futures
> > > are exported
> > > >>>>>> which results in a large number of long lived DGC
> > > threads.  This
> > > >>>>>> turns into a problem for the JVM due to the stack
> > > allocation per
> > > >>>>>> thread.  Presumably this is not good for other reasons
as 
> > > >>>>>> well (e.g., scheduling).
> > > >>>>>>
> > > >>>>>> I have tried to override the leaseValue and checkInterval

> > > >>>>>> defaults per the configuration options below.  I
> > > suspect that the
> > > >>>>>> lease interval is somehow not being obeyed, which
is
> > > presumably a
> > > >>>>>> problem on my end.  However, I can verify that the
> > > configuration
> > > >>>>>> values are in fact showing up in
> > > >>>>>> System.getProperties() for at least some of the 
> JVMs involved 
> > > >>>>>> (the one which drives the workload and the one that
I am 
> > > >>>>>> monitoring with the large number of DGC lease threads).
> > > >>>>>>
> > > >>>>>> Some questions:
> > > >>>>>>
> > > >>>>>> Is this one-thread-per-exported proxy the expected
> > > behavior when
> > > >>>>>> DGC is requested for the exported object?
> > > >>>>>>
> > > >>>>>> The DGC lease checker threads appear to expire ~14
-
> > > 15 minutes
> > > >>>>>> after I terminate the process which was 
> originating the RMI 
> > > >>>>>> requests.  This is close the sum of the default
> > > leaseValue (10m)
> > > >>>>>> and checkInterval (5m) parameters, but maybe there
is
> > > some other
> > > >>>>>> timeout which is controlling this?  If this is the
sum
> > > of those
> > > >>>>>> parameters, why would the DGC lease threads live until
> > > the sum of
> > > >>>>>> those values?  I thought that the lease would expire
after 
> > > >>>>>> the leaseValue (10m default).
> > > >>>>>>
> > > >>>>>> Can the issue I am observing be caused by a low heap
> > > pressure on
> > > >>>>>> the JVM to which the RMI proxies were exported?  If
it
> > > fails to
> > > >>>>>> GC those proxies, even though they are reachable,

> could that 
> > > >>>>>> cause DGC to continue to retain those proxies on the
JVM 
> > > >>>>>> which exported them?
> > > >>>>>>
> > > >>>>>> Is there any way to configure DGC to use a thread
pool
> > > or to have
> > > >>>>>> the leases managed by a single thread?
> > > >>>>>>
> > > >>>>>> Is it possible that there is an interaction with the
> > > useNIO option?
> > > >>>>>>
> > > >>>>>> Relevant options that I am using include:
> > > >>>>>>
> > > >>>>>>   -Dcom.sun.jini.jeri.tcp.useNIO=true
> > > >>>>>>   -Djava.rmi.dgc.leaseValue=30000
> > > >>>>>>   -Dsun.rmi.dgc.checkInterval=15000
> > > >>>>>>   -Dsun.rmi.transport.tcp.connectionPool=true
> > > >>>>>>
> > > >>>>>> Thanks in advance,
> > > >>>>>> Bryan
> > > >>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >
> > > >
> > > >
> > >
> > >
> 
Mime
View raw message