activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Johnny Clark <jcl...@calliduscloud.com>
Subject BrokerService/DefaultMessageListener Shutdown Hanging
Date Fri, 30 May 2014 12:56:50 GMT
*I'm hoping someone will be nice enough to enlighten me as to the proper
way to shutdown my ActiveMQ objects.  I inherited a component that uses
ActiveMQ in conjunction with Spring *DefaultMessageListeners.  The original
author left no information about why various architectural decisions were
made.  I was handed a pretty hot issue a few weeks ago, and after many
hours I feel like I have a basic understanding of the situation.  The
problem was that our application would not shutdown when a "stop" command
was issued from Tomcat.  I resolved the problem, but my solution is a
"hack", (if this is the proper way to do it, I will be shocked).  So now
that I have a little time to breath I'm hoping I can get the correct
solution in place.

I've written a sample application to replicate the problem, and I'll detail
it first.  I'll also include information about my actual application's
architecture and my hack afterwards.  When I create a simple application
with the following content in my Spring XML, then Tomcat is no longer able
to stop my app.  Please note, that I can replicate without ever
sending/receiving a message.  To replicate I simply start the tomcat
process, then log into Tomcat Manager and attempt to execute the stop
command which will never complete.  I originally had the problem in
ActiveMQ 5.8.0 but I've upgrade to 5.9.1.  I'm using Spring 3.0.0, but I
have tried a few minor upgrades on it also.  I can provide the full sample
application if anyone wants it.

I realize that this is a pretty common question.  But I have tried
everything that I can find and nothing has worked.  As I'm sure you know,
ActiveMQ can be used in many different ways.  So I assume the reason I've
had a problem finding a solution is b/c our application must be doing
something a little unusual.  So given the following configuration, how to I
ensure proper shutdown?

  <amq:broker persistent="false" id="mybroker">
    <amq:plugins>
      <amq:statisticsBrokerPlugin/>
    </amq:plugins>
    <amq:transportConnectors>
      <amq:transportConnector uri="tcp://0.0.0.0:61616"/>
    </amq:transportConnectors>
  </amq:broker>

  <amq:connectionFactory id="ConnectionFactory"
brokerURL="vm://localhost?broker.persistent=false" >
    <amq:prefetchPolicy>
      <amq:prefetchPolicy queuePrefetch="0"/>
    </amq:prefetchPolicy>
  </amq:connectionFactory>

  <amq:queue id="lookup.mdb.queue.cat" physicalName="DogQueue"/>
  <amq:queue id="lookup.mdb.queue.dog" physicalName="CatQueue"/>
  <amq:queue id="lookup.mdb.queue.fish" physicalName="FishQueue"/>

  <bean id="messageListener"
class="org.springframework.jms.listener.DefaultMessageListenerContainer"
abstract="true">
    <property name="connectionFactory" ref="ConnectionFactory"/>
  </bean>

  <bean parent="messageListener" id="cat">
    <property name="destination" ref="lookup.mdb.queue.dog"/>
    <property name="messageListener">
      <bean class="com.acteksoft.common.remote.jms.WorkerMessageListener"/>
    </property>
    <property name="concurrentConsumers" value="200"/>
    <property name="maxConcurrentConsumers" value="200"/>
  </bean>

  <bean parent="messageListener" id="dog">
    <property name="destination" ref="lookup.mdb.queue.cat"/>
    <property name="messageListener">
      <bean class="com.acteksoft.common.remote.jms.WorkerMessageListener"/>
    </property>
    <property name="concurrentConsumers" value="200"/>
    <property name="maxConcurrentConsumers" value="200"/>
  </bean>

  <bean parent="messageListener" id="fish">
    <property name="destination" ref="lookup.mdb.queue.fish"/>
    <property name="messageListener">
      <bean class="com.acteksoft.common.remote.jms.WorkerMessageListener"/>
    </property>
    <property name="concurrentConsumers" value="200"/>
    <property name="maxConcurrentConsumers" value="200"/>
  </bean>

If you are interested, here is some info about our system (not the sample
app above) and how we use ActiveMQ.  Our application runs in a
ServletContainer (Tomcat 6/7) and we use Spring for some dependency
injection during start up.  Our system normally consists of several
instances of our application running on multiple servers.  Each instance of
the system knows about the existence of the other instances based on some
content in the DB.  When we have a Job to process we divide up the work and
each server does a portion of the work.  ActiveMQ is used as the medium for
communication between servers while they are processing a "job".  Each
instance of our System has it's own embedded BrokerService running.  Each
instance also has it's own set of queues.  So if the job is started on
System1, then it sends a message to the Job Queue on System2 asking it to
do the first half of the job.  Then System1 will also send a message to the
Job Queue on System3 asking it to do the second half of the job.  Every
queue on every system has it's own DefaultMessageListener.  The Tomcat
Shutdown problem started occurring when they added queuePrefetch="0" to the
connectionFactory bean.  So naturally that was the first thing I
investigated, but after a lot of analysis, I am confident that based on our
architecture the QueuePrefetch needs to be zero.

MY HACK:
Originally we defined the ActiveMQ/DefaultMessageListeners in a spring XML
config file and relied on Spring to shut the objects down properly.  As I
mentioned when they set the QueuePrefetch to zero to resolve another bug,
that is when the system stopped shutting down properly.  My *HACK* consists
of moving the objects out of Spring, creating a ServletContextListener and
then closing the objects manually.  The really hacky part is that I
actually have to spawn another thread to shut down the DefaultMessageListeners.
 If I attempt to shutdown the DefaultMessageListeners before the
Connection/Broker, then the thread hangs, it appears that there is a mutex
waiting for a notify/notifyAll call from another thread and it never
arrives.  If I shutdown the Connection/Broker first, then its a race
situation in which a hang will occur if the Broker shutdown completes
before the DefaultMessageListener shutdown reaches a certain point.  So my
hack involves spawning new threads for each DefaultMessageListener, waiting
a few seconds, allowing them to get to the portion where they enter a wait,
then shutting down the Broker/Connection, which triggers a
notify/notifyAll, which allows the DefaultMessageListener threads to exit
the wait and finish shutting down.  I know it's crazy.

Thank you in advance for any assistance that you can provide.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message