uima-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Satya Nand Kanodia <satya.kano...@orkash.com>
Subject Job gets stuck for infinite time
Date Mon, 23 Mar 2015 06:16:27 GMT
Hi,

In DUCC when I submit a job , sometimes it gets hung for infinite time. 
Today it happened again. A job started  with 2 processes on 22 Mar 2015 
04:34:38   had been continuously  running till today.

I used  this configuration in job file

process_thread_count=3
process_failures_limit=50
process_per_item_time_max=5
process_deployments_max=2
attach_console=true
process_jvm_args=-Xms4g -XX\:+UseConcMarkSweepGC
process_memory_size=7
classpath_order=ducc-before-user

  Finally I had to terminate it from DUCC-MON. Job usually takes 15 
minutes max to process documents. I submitted a batch of 433 documents 
and 342  documents had been processed. There were two errors of cas 
timed out. There were connection exceptions in both processes , which 
lasted for 10 seconds. exception happened at 22 Mar 2015 08:50:42. After 
that no response from both processes.

I got these exceptions from jd.out file

These exception in jd.out file started at 05:09:12.

22 Mar 2015 05:32:12,352 154 ERROR jd.JobDriver waitForProcessStatus     N/A process status
error? java.lang.NullPointerException

At:
	java.util.TreeMap.getEntry(TreeMap.java:342)
	java.util.TreeMap.get(TreeMap.java:273)
	org.apache.uima.ducc.jd.JobDriver.getProcess(JobDriver.java:1081)
	org.apache.uima.ducc.jd.JobDriver.waitForProcessStatus(JobDriver.java:1123)
	org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.java:1585)
	org.apache.uima.ducc.jd.client.WorkItem.exception(WorkItem.java:196)
	org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.java:160)
	java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	java.util.concurrent.FutureTask.run(FutureTask.java:262)
	java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	java.lang.Thread.run(Thread.java:745)

22 Mar 2015 05:32:12,353 154 ERROR jd.JobDriver exception   19524 processing error? java.lang.NullPointerException

At:
	java.util.TreeMap.getEntry(TreeMap.java:342)
	java.util.TreeMap.get(TreeMap.java:273)
	org.apache.uima.ducc.jd.JobDriver.getProcess(JobDriver.java:1081)
	org.apache.uima.ducc.jd.JobDriver.isNodeFailure(JobDriver.java:1092)
	org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.java:1586)
	org.apache.uima.ducc.jd.client.WorkItem.exception(WorkItem.java:196)
	org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.java:160)
	java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	java.util.concurrent.FutureTask.run(FutureTask.java:262)
	java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	java.lang.Thread.run(Thread.java:745)

22 Mar 2015 05:32:12,354 152 ERROR jd.JobDriver waitForProcessStatus     N/A process status
error? java.lang.NullPointerException

At:
	java.util.TreeMap.getEntry(TreeMap.java:342)
	java.util.TreeMap.get(TreeMap.java:273)
	org.apache.uima.ducc.jd.JobDriver.getProcess(JobDriver.java:1081)
	org.apache.uima.ducc.jd.JobDriver.waitForProcessStatus(JobDriver.java:1123)
	org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.java:1585)
	org.apache.uima.ducc.jd.client.WorkItem.exception(WorkItem.java:196)
	org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.java:160)
	java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	java.util.concurrent.FutureTask.run(FutureTask.java:262)
	java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	java.lang.Thread.run(Thread.java:745)

22 Mar 2015 05:32:12,354 152 ERROR jd.JobDriver exception   19524 processing error? java.lang.NullPointerException

At:
	java.util.TreeMap.getEntry(TreeMap.java:342)
	java.util.TreeMap.get(TreeMap.java:273)
	org.apache.uima.ducc.jd.JobDriver.getProcess(JobDriver.java:1081)
	org.apache.uima.ducc.jd.JobDriver.isNodeFailure(JobDriver.java:1092)
	org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.java:1586)
	org.apache.uima.ducc.jd.client.WorkItem.exception(WorkItem.java:196)
	org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.java:160)
	java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	java.util.concurrent.FutureTask.run(FutureTask.java:262)
	java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	java.lang.Thread.run(Thread.java:745)

22 Mar 2015 05:32:12,354 60 ERROR jd.JobDriver waitForProcessStatus     N/A process status
error? java.lang.NullPointerException

At:
	java.util.TreeMap.getEntry(TreeMap.java:342)
	java.util.TreeMap.get(TreeMap.java:273)
	org.apache.uima.ducc.jd.JobDriver.getProcess(JobDriver.java:1081)
	org.apache.uima.ducc.jd.JobDriver.waitForProcessStatus(JobDriver.java:1123)
	org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.java:1585)
	org.apache.uima.ducc.jd.client.WorkItem.exception(WorkItem.java:196)
	org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.java:160)
	java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	java.util.concurrent.FutureTask.run(FutureTask.java:262)
	java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	java.lang.Thread.run(Thread.java:745)

22 Mar 2015 05:32:12,354 60 ERROR jd.JobDriver exception   19524 processing error? java.lang.NullPointerException

At:
	java.util.TreeMap.getEntry(TreeMap.java:342)
	java.util.TreeMap.get(TreeMap.java:273)
	org.apache.uima.ducc.jd.JobDriver.getProcess(JobDriver.java:1081)
	org.apache.uima.ducc.jd.JobDriver.isNodeFailure(JobDriver.java:1092)
	org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.java:1586)
	org.apache.uima.ducc.jd.client.WorkItem.exception(WorkItem.java:196)
	org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.java:160)
	java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	java.util.concurrent.FutureTask.run(FutureTask.java:262)
	java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	java.lang.Thread.run(Thread.java:745)

22 Mar 2015 07:10:59,845 14  WARN jms.DefaultJmsMessageListenerContainer Setup of JMS message
listener invoker failed for destination 'ducc.orchestrator.abbreviated.state' - trying to
recover. Cause: Channel was inactive for too (>30000) long: tcp://192.168.10.144:61617
22 Mar 2015 07:13:09,855 14  WARN jms.DefaultJmsMessageListenerContainer Setup of JMS message
listener invoker failed for destination 'ducc.orchestrator.abbreviated.state' - trying to
recover. Cause: Channel was inactive for too (>30000) long: tcp://192.168.10.144:61617
22 Mar 2015 07:16:50,007 14  WARN jms.DefaultJmsMessageListenerContainer Setup of JMS message
listener invoker failed for destination 'ducc.orchestrator.abbreviated.state' - trying to
recover. Cause: Channel was inactive for too (>30000) long: tcp://192.168.10.144:61617
22 Mar 2015 07:24:35,994 14  WARN jms.EndpointMessageListener Execution of JMS message listener
failed. Caused by: [javax.jms.IllegalStateException - The Session is closed]
javax.jms.IllegalStateException: The Session is closed
	at org.apache.activemq.ActiveMQSession.checkClosed(ActiveMQSession.java:731)
	at org.apache.activemq.ActiveMQSession.getTransacted(ActiveMQSession.java:521)
	at org.apache.activemq.pool.PooledSession.getTransacted(PooledSession.java:256)
	at org.springframework.jms.listener.AbstractMessageListenerContainer.commitIfNecessary(AbstractMessageListenerContainer.java:571)
	at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:481)
	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)
	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)
	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050)
	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
22 Mar 2015 07:24:36,617 14  WARN jms.DefaultJmsMessageListenerContainer Setup of JMS message
listener invoker failed for destination 'ducc.orchestrator.abbreviated.state' - trying to
recover. Cause: The Session is closed
22 Mar 2015 07:26:31,822 14  WARN jms.EndpointMessageListener Execution of JMS message listener
failed. Caused by: [javax.jms.IllegalStateException - The Session is closed]
javax.jms.IllegalStateException: The Session is closed
	at org.apache.activemq.ActiveMQSession.checkClosed(ActiveMQSession.java:731)
	at org.apache.activemq.ActiveMQSession.getTransacted(ActiveMQSession.java:521)
	at org.apache.activemq.pool.PooledSession.getTransacted(PooledSession.java:256)
	at org.springframework.jms.listener.AbstractMessageListenerContainer.commitIfNecessary(AbstractMessageListenerContainer.java:571)
	at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:481)
	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)
	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)
	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050)
	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
22 Mar 2015 07:26:32,067 14  WARN jms.DefaultJmsMessageListenerContainer Setup of JMS message
listener invoker failed for destination 'ducc.orchestrator.abbreviated.state' - trying to
recover. Cause: The Session is closed
22 Mar 2015 07:28:12,075 14  WARN jms.DefaultJmsMessageListenerContainer Setup of JMS message
listener invoker failed for destination 'ducc.orchestrator.abbreviated.state' - trying to
recover. Cause: Channel was inactive for too (>30000) long: tcp://192.168.10.144:61617
22 Mar 2015 07:30:55,001 14  WARN jms.DefaultJmsMessageListenerContainer Could not refresh
JMS Connection for destination 'ducc.orchestrator.abbreviated.state' - retrying in 5000 ms.
Cause: Channel was inactive for too (>30000) long: tcp://192.168.10.144:61617
22 Mar 2015 08:29:26,669 14  WARN jms.EndpointMessageListener Execution of JMS message listener
failed. Caused by: [javax.jms.IllegalStateException - The Session is closed]
javax.jms.IllegalStateException: The Session is closed
	at org.apache.activemq.ActiveMQSession.checkClosed(ActiveMQSession.java:731)
	at org.apache.activemq.ActiveMQSession.getTransacted(ActiveMQSession.java:521)
	at org.apache.activemq.pool.PooledSession.getTransacted(PooledSession.java:256)
	at org.springframework.jms.listener.AbstractMessageListenerContainer.commitIfNecessary(AbstractMessageListenerContainer.java:571)
	at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:481)
	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)
	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)
	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050)
	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
22 Mar 2015 08:29:26,671 14  WARN jms.DefaultJmsMessageListenerContainer Setup of JMS message
listener invoker failed for destination 'ducc.orchestrator.abbreviated.state' - trying to
recover. Cause: The Session is closed
22 Mar 2015 08:33:06,678 14  WARN jms.DefaultJmsMessageListenerContainer Setup of JMS message
listener invoker failed for destination 'ducc.orchestrator.abbreviated.state' - trying to
recover. Cause: Channel was inactive for too (>30000) long: tcp://192.168.10.144:61617
22 Mar 2015 08:36:16,841 14  WARN jms.DefaultJmsMessageListenerContainer Setup of JMS message
listener invoker failed for destination 'ducc.orchestrator.abbreviated.state' - trying to
recover. Cause: Channel was inactive for too (>30000) long: tcp://192.168.10.144:61617
22 Mar 2015 08:37:27,506 14  WARN jms.DefaultJmsMessageListenerContainer Setup of JMS message
listener invoker failed for destination 'ducc.orchestrator.abbreviated.state' - trying to
recover. Cause: Channel was inactive for too (>30000) long: tcp://192.168.10.144:61617
22 Mar 2015 08:39:07,876 14  WARN jms.DefaultJmsMessageListenerContainer Setup of JMS message
listener invoker failed for destination 'ducc.orchestrator.abbreviated.state' - trying to
recover. Cause: Channel was inactive for too (>30000) long: tcp://192.168.10.144:61617
22 Mar 2015 08:42:08,029 14  WARN jms.DefaultJmsMessageListenerContainer Setup of JMS message
listener invoker failed for destination 'ducc.orchestrator.abbreviated.state' - trying to
recover. Cause: Channel was inactive for too (>30000) long: tcp://192.168.10.144:61617
22 Mar 2015 08:42:39,341 15 ERROR processor.DefaultErrorHandler Failed delivery for (MessageId:
ID-S143-52155-1426979058276-0-2895 on ExchangeId: ID-S143-52155-1426979058276-0-2896). Exhausted
after delivery attempt: 1 caught: org.springframework.jms.UncategorizedJmsException: Uncategorized
exception occured during JMS processing; nested exception is javax.jms.JMSException: Channel
was inactive for too (>30000) long: tcp://192.168.10.144:61617
org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during
JMS processing; nested exception is javax.jms.JMSException: Channel was inactive for too (>30000)
long: tcp://192.168.10.144:61617
	at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)
	at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:168)
	at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:469)
	at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.send(JmsConfiguration.java:188)
	at org.apache.camel.component.jms.JmsProducer.doSend(JmsProducer.java:398)
	at org.apache.camel.component.jms.JmsProducer.processInOnly(JmsProducer.java:352)
	at org.apache.camel.component.jms.JmsProducer.process(JmsProducer.java:132)
	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
	at org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:120)
	at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:292)
	at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:115)
	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
	at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
	at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:73)
	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
	at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
	at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91)
	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
	at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:330)
	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:220)
	at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
	at org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:303)
	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:117)
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
	at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
	at org.apache.camel.processor.UnitOfWorkProcessor.processAsync(UnitOfWorkProcessor.java:150)
	at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:117)
	at org.apache.camel.processor.RouteInflightRepositoryProcessor.processNext(RouteInflightRepositoryProcessor.java:48)
	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
	at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
	at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:73)
	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:99)
	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:86)
	at org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:134)
	at org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:63)
	at java.util.TimerThread.mainLoop(Timer.java:555)
	at java.util.TimerThread.run(Timer.java:505)
Caused by: javax.jms.JMSException: Channel was inactive for too (>30000) long: tcp://192.168.10.144:61617
	at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
	at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1377)
	at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1305)
	at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1795)
	at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:277)
	at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:212)
	at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:79)
	at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:62)
	at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:592)
	at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.doSend(JmsConfiguration.java:276)
	at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.doSendToDestination(JmsConfiguration.java:215)
	at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.access$100(JmsConfiguration.java:157)
	at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate$3.doInJms(JmsConfiguration.java:191)
	at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:466)
	... 41 more
Caused by: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too
(>30000) long: tcp://192.168.10.144:61617
	at org.apache.activemq.transport.AbstractInactivityMonitor$4.run(AbstractInactivityMonitor.java:209)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
22 Mar 2015 08:42:39,546 15  WARN timer.TimerConsumer Error processing exchange. Exchange[Message:
org.apache.uima.ducc.transport.event.JdStateDuccEvent@db25b23]. Caused by: [org.springframework.jms.UncategorizedJmsException
- Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException:
Channel was inactive for too (>30000) long: tcp://192.168.10.144:61617]
org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during
JMS processing; nested exception is javax.jms.JMSException: Channel was inactive for too (>30000)
long: tcp://192.168.10.144:61617
	at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)
	at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:168)
	at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:469)
	at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.send(JmsConfiguration.java:188)
	at org.apache.camel.component.jms.JmsProducer.doSend(JmsProducer.java:398)
	at org.apache.camel.component.jms.JmsProducer.processInOnly(JmsProducer.java:352)
	at org.apache.camel.component.jms.JmsProducer.process(JmsProducer.java:132)
	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
	at org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:120)
	at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:292)
	at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:115)
	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
	at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
	at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:73)
	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
	at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
	at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91)
	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
	at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:330)
	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:220)
	at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
	at org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:303)
	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:117)
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
	at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
	at org.apache.camel.processor.UnitOfWorkProcessor.processAsync(UnitOfWorkProcessor.java:150)
	at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:117)
	at org.apache.camel.processor.RouteInflightRepositoryProcessor.processNext(RouteInflightRepositoryProcessor.java:48)
	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
	at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
	at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:73)
	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:99)
	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:86)
	at org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:134)
	at org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:63)
	at java.util.TimerThread.mainLoop(Timer.java:555)
	at java.util.TimerThread.run(Timer.java:505)
Caused by: javax.jms.JMSException: Channel was inactive for too (>30000) long: tcp://192.168.10.144:61617
	at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
	at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1377)
	at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1305)
	at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1795)
	at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:277)
	at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:212)
	at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:79)
	at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:62)
	at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:592)
	at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.doSend(JmsConfiguration.java:276)
	at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.doSendToDestination(JmsConfiguration.java:215)
	at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.access$100(JmsConfiguration.java:157)
	at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate$3.doInJms(JmsConfiguration.java:191)
	at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:466)
	... 41 more
Caused by: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too
(>30000) long: tcp://192.168.10.144:61617
	at org.apache.activemq.transport.AbstractInactivityMonitor$4.run(AbstractInactivityMonitor.java:209)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
22 Mar 2015 08:43:19,482 14  WARN jms.DefaultJmsMessageListenerContainer Setup of JMS message
listener invoker failed for destination 'ducc.orchestrator.abbreviated.state' - trying to
recover. Cause: Channel was inactive for too (>30000) long: tcp://192.168.10.144:61617
22 Mar 2015 08:44:59,642 14  WARN jms.DefaultJmsMessageListenerContainer Setup of JMS message
listener invoker failed for destination 'ducc.orchestrator.abbreviated.state' - trying to
recover. Cause: Channel was inactive for too (>30000) long: tcp://192.168.10.144:61617
22 Mar 2015 08:46:10,729 14  WARN jms.DefaultJmsMessageListenerContainer Setup of JMS message
listener invoker failed for destination 'ducc.orchestrator.abbreviated.state' - trying to
recover. Cause: Channel was inactive for too (>30000) long: tcp://192.168.10.144:61617
22 Mar 2015 08:48:05,731 14  WARN jms.DefaultJmsMessageListenerContainer Setup of JMS message
listener invoker failed for destination 'ducc.orchestrator.abbreviated.state' - trying to
recover. Cause: Channel was inactive for too (>30000) long: tcp://192.168.10.144:61617
22 Mar 2015 08:49:49,723 14  WARN jms.DefaultJmsMessageListenerContainer Setup of JMS message
listener invoker failed for destination 'ducc.orchestrator.abbreviated.state' - trying to
recover. Cause: Channel was inactive for too (>30000) long: tcp://192.168.10.144:61617



Please help me to find out root cause of this and how can I kill a job 
or start a new one when any job gets stuck after a certain time..

-- 
Thanks and Regards,
Satya Nand Kanodia


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message