activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <claus.ib...@gmail.com>
Subject Re: [HEADS UP] - Graceful shutdown of AMQ and fixing some leaks when doing hot-deployment
Date Fri, 07 Sep 2012 06:33:49 GMT
On Thu, Sep 6, 2012 at 7:29 PM, Claudio Corsi <clcorsi@yahoo.com> wrote:
> Claus,
>
> Why did you not consider using the keep alive option of the thread pool?  This would
reclaim threads that have been idle after a given certain time and those resources would of
been reclaimed.   The only consideration is that you would need to reduce the number of active
threads to a small value since this is useful for created threads that exceed the minimum
thread pool size.
>

The thread pools in AMQ do have keep alive times, usually set to 30
sec, and some have lower values such as 10 sec etc.
And any of the patches/commits do not change how these thread pools is
configured out of the box.

Its the fact the old code uses a shared static thread pool (for many
of its async tasks related to stopping/closing), and that thread pool
is newer shutdown.
Which leaves the thread pool around as well its threads. And as well
some thread pools wasn't shutdown either (which could potential leave
threads around, until they terminate). And btw some thread pools such
as the scheduled thread pool (AFAIR this is how they have been
implemented in the JDK) never terminate, so its important to shutdown
these manually.

Also the goal when shutting down is to give all active and running
threads a chance to complete. And if not, then become more aggressive
and shutdown them using interrupts etc. In the end when your app with
embedded AMQ Client (or broker) is shutdown and being undeployed, the
goal is to not have any threads or leaks running around.

As you have looked in the JIRA ticket about that "Tomcat report" you
have seen that it helps fixing this by shutting down that shared
thread pool.
So we are on a path for a solution. Just taking a bit more time, to
have a common API to shutdown thread pools in AMQ. Then we can better
"control" this.



> --Claudio
>
>
>>________________________________
>> From: Claus Ibsen <claus.ibsen@gmail.com>
>>To: dev@activemq.apache.org
>>Sent: Thursday, September 6, 2012 11:20 AM
>>Subject: Re: [HEADS UP] - Graceful shutdown of AMQ and fixing some leaks when doing
hot-deployment
>>
>>Hi
>>
>>Okay I have been running a bunch of unit tests, and I can see that the
>>tests from activemq-core, is kinda dependent on the old behavior
>>of shutting down thread pools aggressively using shutdownNow or no
>>attempt to wait for in progress tasks to complete gracefully.
>>
>>So the changes being committed is currently to keep same behavior of
>>shutting down. Then we can gently switch over to
>>graceful shutdown as we fix the tests and identify the ones to switch over.
>>
>>So ThreadPoolUtils have 3 set of shutdown methods
>>- shutdown
>>- shutdownNow
>>- shutdownGraceful
>>
>>The first 2 is just like on the JDK executor service.
>>The last one is the graceful one.
>>
>>
>>
>>On Thu, Sep 6, 2012 at 2:47 PM, Claus Ibsen <claus.ibsen@gmail.com> wrote:
>>> Hi
>>>
>>> Recently I have been diving into fixing some leaks in ActiveMQ. It
>>> started with the activemq-pool component which caused a Camel JMS
>>> client to become slower, due memory leaks not being released.
>>>
>>> The activemq-pool has been fixed now, and runs flawless again.
>>> On a side note Tim Bish is working on improving the activemq-pool to
>>> be a better pool by leveraging more logic from commons-pool on the
>>> connection side as well. Beforehand it was only the sessions that was
>>> leveraged.
>>>
>>> Anyway continuing with the Camel JMS client, we have reports, about
>>> leaks in Tomcat, when you redeploy an application. So I have been
>>> tracking those down, and got most of it fixed. Have some pending
>>> commits for the JMS client.
>>>
>>> However on embedded a full blow ActiveMQ broker in a web container as
>>> Tomcat, and then supporting hot deployment, surfaced some issues with
>>> some thread pools not being closed graceful. Instead the pools is
>>> closed aggresively, and as some of them have idle threads timeout of
>>> 30 seconds, then those threads dont timeout before the application is
>>> redeployed (assuming you re-deploy the app; to remedy this you would
>>> have to stop the app, and wait for > 30 sec, and then deploy again).
>>>
>>> As we have some good graceful shutdown code in Camel, I have ported
>>> part of that to AMQ, in the ThreadPoolUtils class in the util package.
>>> There is a number of shutdown methods, to use for graceful shutdown.
>>> When a graceful shutdown of a thread pool is in the works, and if it
>>> takes some times, then we now log every 5th second about the shutdown
>>> in progress.
>>>
>>>
>>> There is a number of tickets in JIRA about this recent work (all the
>>> stuff above, etc.)
>>> https://issues.apache.org/jira/browse/AMQ-4026
>>> https://issues.apache.org/jira/browse/AMQ-4019
>>> https://issues.apache.org/jira/browse/AMQ-3451
>>> https://issues.apache.org/jira/browse/AMQ-4016
>>> https://issues.apache.org/jira/browse/AMQ-3959
>>> https://issues.apache.org/jira/browse/AMQ-3797
>>> https://issues.apache.org/jira/browse/AMQ-4008
>>> https://issues.apache.org/jira/browse/AMQ-4011
>>> https://issues.apache.org/jira/browse/AMQ-3997
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> -----------------
>>> FuseSource
>>> Email: cibsen@fusesource.com
>>> Web: http://fusesource.com
>>> Twitter: davsclaus, fusenews
>>> Blog: http://davsclaus.com
>>> Author of Camel in Action: http://www.manning.com/ibsen
>>
>>
>>
>>--
>>Claus Ibsen
>>-----------------
>>FuseSource
>>Email: cibsen@fusesource.com
>>Web: http://fusesource.com
>>Twitter: davsclaus, fusenews
>>Blog: http://davsclaus.com
>>Author of Camel in Action: http://www.manning.com/ibsen
>>
>>
>>



-- 
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen

Mime
View raw message