camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Lamb <dan...@discoverygarden.ca>
Subject Interrupted Exception when Shutting Down with Embedded Activemq
Date Mon, 30 Mar 2015 14:26:22 GMT
Hi all,

I’m running a stand-alone camel app with an embedded activemq broker and am running into
unclean shutdowns.  When shutting down, I get a graceful shutdown of the camel context, and
then an InterruptedExcpetion from KahaDB when ActiveMQ attempts shutdown afterwards.

[                      Thread-6] MainSupport                    INFO  Apache Camel 2.14.2
stopping
[                      Thread-6] SpringCamelContext             INFO  Apache Camel 2.14.2
(CamelContext: camel) is shutting down
[                      Thread-6] DefaultShutdownStrategy        INFO  Starting to graceful
shutdown 2 routes (timeout 300 seconds)
[amel) thread #2 - ShutdownTask] DefaultShutdownStrategy        INFO  Route: route2 shutdown
complete, was consuming from: Endpoint[file:///home/vagrant/fedora3data/objectStore?noop=true&recursive=true&sendEmptyMessageWhenIdle=true]
[amel) thread #2 - ShutdownTask] DefaultShutdownStrategy        INFO  Route: route1 shutdown
complete, was consuming from: Endpoint[activemq://foxml]
[                      Thread-6] DefaultShutdownStrategy        INFO  Graceful shutdown of
2 routes completed in 0 seconds
[                      Thread-6] SpringCamelContext             INFO  Apache Camel 2.14.2
(CamelContext: camel) uptime 3.128 seconds
[                      Thread-6] SpringCamelContext             INFO  Apache Camel 2.14.2
(CamelContext: camel) is shutdown in 0.494 seconds
[                      Thread-6] BrokerService                  INFO  Apache ActiveMQ 5.11.1
(myBroker, ID:islandora-51066-1427724041540-0:1) is shutting down
[                      Thread-6] TransportConnector             INFO  Connector vm stopped
[                      Thread-6] PListStoreImpl                 INFO  PListStore:[/home/vagrant/spring/migration-utils/src/data/myBroker/tmp_storage]
stopped
[                      Thread-6] KahaDBStore                    INFO  Stopping async queue
tasks
[                      Thread-6] KahaDBStore                    ERROR Could not stop service:
KahaDB:[/home/vagrant/spring/migration-utils/src/data/myBroker/KahaDB]. Reason: java.lang.InterruptedException
java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325)
	at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:588)
	at org.apache.activemq.store.kahadb.KahaDBStore.doStop(KahaDBStore.java:247)
	at org.apache.activemq.util.ServiceSupport.stop(ServiceSupport.java:71)
	at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.doStop(KahaDBPersistenceAdapter.java:250)
	at org.apache.activemq.util.ServiceSupport.stop(ServiceSupport.java:71)
	at org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41)
	at org.apache.activemq.broker.BrokerService.stop(BrokerService.java:792)
	at org.apache.activemq.xbean.XBeanBrokerService.stop(XBeanBrokerService.java:122)
	at org.apache.activemq.xbean.XBeanBrokerService.destroy(XBeanBrokerService.java:115)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:350)
	at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:273)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:538)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:514)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:831)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:483)
	at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1092)
	at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1066)
	at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1012)
	at org.apache.camel.util.IOHelper.close(IOHelper.java:326)
	at org.apache.camel.util.IOHelper.close(IOHelper.java:390)
	at org.apache.camel.spring.Main.doStop(Main.java:183)
	at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:102)
	at org.fcrepo.migration.processors.ShutdownProcessor$1.run(ShutdownProcessor.java:23)


The shutdown does eventually continue and complete after hanging for a bit, with the following
message:

[                      Thread-6] BrokerService                  INFO  Apache ActiveMQ 5.11.1
(myBroker, ID:islandora-51066-1427724041540-0:1) uptime 4.065 seconds
[                      Thread-6] BrokerService                  INFO  Apache ActiveMQ 5.11.1
(myBroker, ID:islandora-51066-1427724041540-0:1) is shutdown
[                      Thread-6] DisposableBeanAdapter          WARN  Invocation of destroy
method 'destroy' failed on bean with name 'broker': java.lang.InterruptedException
[WARNING] thread Thread[KahaDB Scheduler,5,org.apache.camel.spring.Main] was interrupted but
is still alive after waiting at least 14995msecs
[WARNING] thread Thread[KahaDB Scheduler,5,org.apache.camel.spring.Main] will linger despite
being asked to die via interruption
[WARNING] thread Thread[ConcurrentQueueStoreAndDispatch,5,org.apache.camel.spring.Main] will
linger despite being asked to die via interruption
[WARNING] NOTE: 2 thread(s) did not finish despite being asked to  via interruption. This
is not a problem with exec:java, it is a problem with the running code. Although not serious,
it should be remedied.
[WARNING] Couldn't destroy threadgroup org.apache.camel.maven.RunMojo$IsolatedThreadGroup[name=org.apache.camel.spring.Main,maxpri=10]
java.lang.IllegalThreadStateException

The interrupted exception makes sense.  I’m just wondering what’s the best approach to
handling this properly?  I’m shutting down the application using another thread as per http://camel.apache.org/how-can-i-stop-a-route-from-a-route.html.
 Is the problem perhaps how I’m shutting down the Spring context by calling stop on Main?
 Is there a better way to stop a command-line camel app to ensure things get handled in order
and wait for the broker and persistence services to shutdown cleanly?

I’ve personally tried shutting down the camel context manually before the spring context,
as well as shutdown hooks on the broker bean.  Nothing seems to be working.  Here’s my shutdown
code:

package org.fcrepo.migration.processors;

import org.apache.camel.Exchange;
import org.apache.camel.Processor;
import org.apache.camel.spring.Main;

/**
 * Shuts down the application.
 *
 * @author Daniel Lamb
 */
public class ShutdownProcessor implements Processor {

    private Thread stop;

    @Override
    public void process(final Exchange exchange) throws Exception {
        if (stop == null) {
            stop = new Thread() {
                @Override
                public void run() {
                    try {
                        Main.getInstance().stop();
                    }
                    catch (Exception e) {
                        // ignore
                    }
                }
            };
        }

        stop.start();
    }
}

Here’s my configs (pretty much ripped straight out of the camel activemq-tomcat example:

Camel:
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:amq="http://activemq.apache.org/schema/core"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:p="http://www.springframework.org/schema/p"
       xsi:schemaLocation="
       http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
       http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd">

  <!-- config -->
  <bean id="bridgePropertyPlaceholder"
        class="org.apache.camel.spring.spi.BridgePropertyPlaceholderConfigurer">
      <property name="location" value="classpath:migration.properties"/>
  </bean>

  <!-- activemq -->
  <bean id="activemq"
        class="org.apache.activemq.camel.component.ActiveMQComponent">
      <property name="brokerURL" value="vm://myBroker?create=false&amp;waitForStart=5000"/>
  </bean>

  <camelContext xmlns="http://camel.apache.org/schema/spring"
                id="camel"
                shutdownRunningTask="CompleteAllTasks">
      <package>org.fcrepo.migration.routes</package>
  </camelContext>

</beans>

Activemq:
<beans
  xmlns="http://www.springframework.org/schema/beans"
  xmlns:context="http://www.springframework.org/schema/context"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
  http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd
  http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">

  <broker xmlns="http://activemq.apache.org/schema/core"
          id="broker"
          brokerName="myBroker"
          useShutdownHook=“false"
          persistent="true"
          dataDirectory="${activemqData}">

      <transportConnectors>
          <transportConnector name="vm" uri="vm://myBroker"/>
      </transportConnectors>
  </broker>
</beans>

Any advice/experience would be greatly appreciated.

~Danny

  
Mime
View raw message