uima-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lou DeGenaro <lou.degen...@gmail.com>
Subject Re: Job gets stuck for infinite time
Date Mon, 23 Mar 2015 12:56:00 GMT
A Jira already exists for this issue https://issues.apache.org/jira/browse/
UIMA-4225.  It has been marked "Won't Fix" because the Job Driver and
corresponding Job Process have been re-written for 2.0 version of DUCC (not
yet released, but available from trunk).

Are you able to reproduce this error with regularity?  I've only seen it
once before, which resulted in Jira 4225 and its subsequent closure. It
also seems that you may have more serious problems than just one job
failing.  Based on the message:

22 Mar 2015 08:49:49,723 14  WARN jms.DefaultJmsMessageListenerC
ontainer Setup of JMS message listener invoker failed for destination
'ducc.orchestrator.abbreviated.state' - trying to recover.

it seems you may have network or broker problems?

Lou.


On Mon, Mar 23, 2015 at 2:16 AM, Satya Nand Kanodia <
satya.kanodia@orkash.com> wrote:

> Hi,
>
> In DUCC when I submit a job , sometimes it gets hung for infinite time.
> Today it happened again. A job started  with 2 processes on 22 Mar 2015
> 04:34:38   had been continuously  running till today.
>
> I used  this configuration in job file
>
> process_thread_count=3
> process_failures_limit=50
> process_per_item_time_max=5
> process_deployments_max=2
> attach_console=true
> process_jvm_args=-Xms4g -XX\:+UseConcMarkSweepGC
> process_memory_size=7
> classpath_order=ducc-before-user
>
>  Finally I had to terminate it from DUCC-MON. Job usually takes 15 minutes
> max to process documents. I submitted a batch of 433 documents and 342
> documents had been processed. There were two errors of cas timed out. There
> were connection exceptions in both processes , which lasted for 10 seconds.
> exception happened at 22 Mar 2015 08:50:42. After that no response from
> both processes.
>
> I got these exceptions from jd.out file
>
> These exception in jd.out file started at 05:09:12.
>
> 22 Mar 2015 05:32:12,352 154 ERROR jd.JobDriver waitForProcessStatus
>  N/A process status error? java.lang.NullPointerException
>
> At:
>         java.util.TreeMap.getEntry(TreeMap.java:342)
>         java.util.TreeMap.get(TreeMap.java:273)
>         org.apache.uima.ducc.jd.JobDriver.getProcess(JobDriver.java:1081)
>         org.apache.uima.ducc.jd.JobDriver.waitForProcessStatus(
> JobDriver.java:1123)
>         org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.java:1585)
>         org.apache.uima.ducc.jd.client.WorkItem.exception(
> WorkItem.java:196)
>         org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.java:160)
>         java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:471)
>         java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
>         java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
>         java.lang.Thread.run(Thread.java:745)
>
> 22 Mar 2015 05:32:12,353 154 ERROR jd.JobDriver exception   19524
> processing error? java.lang.NullPointerException
>
> At:
>         java.util.TreeMap.getEntry(TreeMap.java:342)
>         java.util.TreeMap.get(TreeMap.java:273)
>         org.apache.uima.ducc.jd.JobDriver.getProcess(JobDriver.java:1081)
>         org.apache.uima.ducc.jd.JobDriver.isNodeFailure(
> JobDriver.java:1092)
>         org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.java:1586)
>         org.apache.uima.ducc.jd.client.WorkItem.exception(
> WorkItem.java:196)
>         org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.java:160)
>         java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:471)
>         java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
>         java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
>         java.lang.Thread.run(Thread.java:745)
>
> 22 Mar 2015 05:32:12,354 152 ERROR jd.JobDriver waitForProcessStatus
>  N/A process status error? java.lang.NullPointerException
>
> At:
>         java.util.TreeMap.getEntry(TreeMap.java:342)
>         java.util.TreeMap.get(TreeMap.java:273)
>         org.apache.uima.ducc.jd.JobDriver.getProcess(JobDriver.java:1081)
>         org.apache.uima.ducc.jd.JobDriver.waitForProcessStatus(
> JobDriver.java:1123)
>         org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.java:1585)
>         org.apache.uima.ducc.jd.client.WorkItem.exception(
> WorkItem.java:196)
>         org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.java:160)
>         java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:471)
>         java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
>         java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
>         java.lang.Thread.run(Thread.java:745)
>
> 22 Mar 2015 05:32:12,354 152 ERROR jd.JobDriver exception   19524
> processing error? java.lang.NullPointerException
>
> At:
>         java.util.TreeMap.getEntry(TreeMap.java:342)
>         java.util.TreeMap.get(TreeMap.java:273)
>         org.apache.uima.ducc.jd.JobDriver.getProcess(JobDriver.java:1081)
>         org.apache.uima.ducc.jd.JobDriver.isNodeFailure(
> JobDriver.java:1092)
>         org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.java:1586)
>         org.apache.uima.ducc.jd.client.WorkItem.exception(
> WorkItem.java:196)
>         org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.java:160)
>         java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:471)
>         java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
>         java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
>         java.lang.Thread.run(Thread.java:745)
>
> 22 Mar 2015 05:32:12,354 60 ERROR jd.JobDriver waitForProcessStatus
>  N/A process status error? java.lang.NullPointerException
>
> At:
>         java.util.TreeMap.getEntry(TreeMap.java:342)
>         java.util.TreeMap.get(TreeMap.java:273)
>         org.apache.uima.ducc.jd.JobDriver.getProcess(JobDriver.java:1081)
>         org.apache.uima.ducc.jd.JobDriver.waitForProcessStatus(
> JobDriver.java:1123)
>         org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.java:1585)
>         org.apache.uima.ducc.jd.client.WorkItem.exception(
> WorkItem.java:196)
>         org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.java:160)
>         java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:471)
>         java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
>         java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
>         java.lang.Thread.run(Thread.java:745)
>
> 22 Mar 2015 05:32:12,354 60 ERROR jd.JobDriver exception   19524
> processing error? java.lang.NullPointerException
>
> At:
>         java.util.TreeMap.getEntry(TreeMap.java:342)
>         java.util.TreeMap.get(TreeMap.java:273)
>         org.apache.uima.ducc.jd.JobDriver.getProcess(JobDriver.java:1081)
>         org.apache.uima.ducc.jd.JobDriver.isNodeFailure(
> JobDriver.java:1092)
>         org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.java:1586)
>         org.apache.uima.ducc.jd.client.WorkItem.exception(
> WorkItem.java:196)
>         org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.java:160)
>         java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:471)
>         java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
>         java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
>         java.lang.Thread.run(Thread.java:745)
>
> 22 Mar 2015 07:10:59,845 14  WARN jms.DefaultJmsMessageListenerContainer
> Setup of JMS message listener invoker failed for destination
> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel
> was inactive for too (>30000) long: tcp://192.168.10.144:61617
> 22 Mar 2015 07:13:09,855 14  WARN jms.DefaultJmsMessageListenerContainer
> Setup of JMS message listener invoker failed for destination
> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel
> was inactive for too (>30000) long: tcp://192.168.10.144:61617
> 22 Mar 2015 07:16:50,007 14  WARN jms.DefaultJmsMessageListenerContainer
> Setup of JMS message listener invoker failed for destination
> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel
> was inactive for too (>30000) long: tcp://192.168.10.144:61617
> 22 Mar 2015 07:24:35,994 14  WARN jms.EndpointMessageListener Execution of
> JMS message listener failed. Caused by: [javax.jms.IllegalStateException
> - The Session is closed]
> javax.jms.IllegalStateException: The Session is closed
>         at org.apache.activemq.ActiveMQSession.checkClosed(
> ActiveMQSession.java:731)
>         at org.apache.activemq.ActiveMQSession.getTransacted(
> ActiveMQSession.java:521)
>         at org.apache.activemq.pool.PooledSession.getTransacted(
> PooledSession.java:256)
>         at org.springframework.jms.listener.AbstractMessageListenerContain
> er.commitIfNecessary(AbstractMessageListenerContainer.java:571)
>         at org.springframework.jms.listener.AbstractMessageListenerContain
> er.doExecuteListener(AbstractMessageListenerContainer.java:481)
>         at org.springframework.jms.listener.AbstractPollingMessageListener
> Container.doReceiveAndExecute(AbstractPollingMessageListener
> Container.java:325)
>         at org.springframework.jms.listener.AbstractPollingMessageListener
> Container.receiveAndExecute(AbstractPollingMessageListener
> Container.java:263)
>         at org.springframework.jms.listener.DefaultMessageListenerContaine
> r$AsyncMessageListenerInvoker.invokeListener(
> DefaultMessageListenerContainer.java:1058)
>         at org.springframework.jms.listener.DefaultMessageListenerContaine
> r$AsyncMessageListenerInvoker.executeOngoingLoop(
> DefaultMessageListenerContainer.java:1050)
>         at org.springframework.jms.listener.DefaultMessageListenerContaine
> r$AsyncMessageListenerInvoker.run(DefaultMessageListenerContaine
> r.java:947)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> 22 Mar 2015 07:24:36,617 14  WARN jms.DefaultJmsMessageListenerContainer
> Setup of JMS message listener invoker failed for destination
> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: The
> Session is closed
> 22 Mar 2015 07:26:31,822 14  WARN jms.EndpointMessageListener Execution of
> JMS message listener failed. Caused by: [javax.jms.IllegalStateException
> - The Session is closed]
> javax.jms.IllegalStateException: The Session is closed
>         at org.apache.activemq.ActiveMQSession.checkClosed(
> ActiveMQSession.java:731)
>         at org.apache.activemq.ActiveMQSession.getTransacted(
> ActiveMQSession.java:521)
>         at org.apache.activemq.pool.PooledSession.getTransacted(
> PooledSession.java:256)
>         at org.springframework.jms.listener.AbstractMessageListenerContain
> er.commitIfNecessary(AbstractMessageListenerContainer.java:571)
>         at org.springframework.jms.listener.AbstractMessageListenerContain
> er.doExecuteListener(AbstractMessageListenerContainer.java:481)
>         at org.springframework.jms.listener.AbstractPollingMessageListener
> Container.doReceiveAndExecute(AbstractPollingMessageListener
> Container.java:325)
>         at org.springframework.jms.listener.AbstractPollingMessageListener
> Container.receiveAndExecute(AbstractPollingMessageListener
> Container.java:263)
>         at org.springframework.jms.listener.DefaultMessageListenerContaine
> r$AsyncMessageListenerInvoker.invokeListener(
> DefaultMessageListenerContainer.java:1058)
>         at org.springframework.jms.listener.DefaultMessageListenerContaine
> r$AsyncMessageListenerInvoker.executeOngoingLoop(
> DefaultMessageListenerContainer.java:1050)
>         at org.springframework.jms.listener.DefaultMessageListenerContaine
> r$AsyncMessageListenerInvoker.run(DefaultMessageListenerContaine
> r.java:947)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> 22 Mar 2015 07:26:32,067 14  WARN jms.DefaultJmsMessageListenerContainer
> Setup of JMS message listener invoker failed for destination
> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: The
> Session is closed
> 22 Mar 2015 07:28:12,075 14  WARN jms.DefaultJmsMessageListenerContainer
> Setup of JMS message listener invoker failed for destination
> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel
> was inactive for too (>30000) long: tcp://192.168.10.144:61617
> 22 Mar 2015 07:30:55,001 14  WARN jms.DefaultJmsMessageListenerContainer
> Could not refresh JMS Connection for destination 'ducc.orchestrator.abbreviated.state'
> - retrying in 5000 ms. Cause: Channel was inactive for too (>30000) long:
> tcp://192.168.10.144:61617
> 22 Mar 2015 08:29:26,669 14  WARN jms.EndpointMessageListener Execution of
> JMS message listener failed. Caused by: [javax.jms.IllegalStateException
> - The Session is closed]
> javax.jms.IllegalStateException: The Session is closed
>         at org.apache.activemq.ActiveMQSession.checkClosed(
> ActiveMQSession.java:731)
>         at org.apache.activemq.ActiveMQSession.getTransacted(
> ActiveMQSession.java:521)
>         at org.apache.activemq.pool.PooledSession.getTransacted(
> PooledSession.java:256)
>         at org.springframework.jms.listener.AbstractMessageListenerContain
> er.commitIfNecessary(AbstractMessageListenerContainer.java:571)
>         at org.springframework.jms.listener.AbstractMessageListenerContain
> er.doExecuteListener(AbstractMessageListenerContainer.java:481)
>         at org.springframework.jms.listener.AbstractPollingMessageListener
> Container.doReceiveAndExecute(AbstractPollingMessageListener
> Container.java:325)
>         at org.springframework.jms.listener.AbstractPollingMessageListener
> Container.receiveAndExecute(AbstractPollingMessageListener
> Container.java:263)
>         at org.springframework.jms.listener.DefaultMessageListenerContaine
> r$AsyncMessageListenerInvoker.invokeListener(
> DefaultMessageListenerContainer.java:1058)
>         at org.springframework.jms.listener.DefaultMessageListenerContaine
> r$AsyncMessageListenerInvoker.executeOngoingLoop(
> DefaultMessageListenerContainer.java:1050)
>         at org.springframework.jms.listener.DefaultMessageListenerContaine
> r$AsyncMessageListenerInvoker.run(DefaultMessageListenerContaine
> r.java:947)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> 22 Mar 2015 08:29:26,671 14  WARN jms.DefaultJmsMessageListenerContainer
> Setup of JMS message listener invoker failed for destination
> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: The
> Session is closed
> 22 Mar 2015 08:33:06,678 14  WARN jms.DefaultJmsMessageListenerContainer
> Setup of JMS message listener invoker failed for destination
> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel
> was inactive for too (>30000) long: tcp://192.168.10.144:61617
> 22 Mar 2015 08:36:16,841 14  WARN jms.DefaultJmsMessageListenerContainer
> Setup of JMS message listener invoker failed for destination
> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel
> was inactive for too (>30000) long: tcp://192.168.10.144:61617
> 22 Mar 2015 08:37:27,506 14  WARN jms.DefaultJmsMessageListenerContainer
> Setup of JMS message listener invoker failed for destination
> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel
> was inactive for too (>30000) long: tcp://192.168.10.144:61617
> 22 Mar 2015 08:39:07,876 14  WARN jms.DefaultJmsMessageListenerContainer
> Setup of JMS message listener invoker failed for destination
> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel
> was inactive for too (>30000) long: tcp://192.168.10.144:61617
> 22 Mar 2015 08:42:08,029 14  WARN jms.DefaultJmsMessageListenerContainer
> Setup of JMS message listener invoker failed for destination
> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel
> was inactive for too (>30000) long: tcp://192.168.10.144:61617
> 22 Mar 2015 08:42:39,341 15 ERROR processor.DefaultErrorHandler Failed
> delivery for (MessageId: ID-S143-52155-1426979058276-0-2895 on
> ExchangeId: ID-S143-52155-1426979058276-0-2896). Exhausted after delivery
> attempt: 1 caught: org.springframework.jms.UncategorizedJmsException:
> Uncategorized exception occured during JMS processing; nested exception is
> javax.jms.JMSException: Channel was inactive for too (>30000) long: tcp://
> 192.168.10.144:61617
> org.springframework.jms.UncategorizedJmsException: Uncategorized
> exception occured during JMS processing; nested exception is
> javax.jms.JMSException: Channel was inactive for too (>30000) long: tcp://
> 192.168.10.144:61617
>         at org.springframework.jms.support.JmsUtils.
> convertJmsAccessException(JmsUtils.java:316)
>         at org.springframework.jms.support.JmsAccessor.
> convertJmsAccessException(JmsAccessor.java:168)
>         at org.springframework.jms.core.JmsTemplate.execute(
> JmsTemplate.java:469)
>         at org.apache.camel.component.jms.JmsConfiguration$
> CamelJmsTemplate.send(JmsConfiguration.java:188)
>         at org.apache.camel.component.jms.JmsProducer.doSend(
> JmsProducer.java:398)
>         at org.apache.camel.component.jms.JmsProducer.processInOnly(
> JmsProducer.java:352)
>         at org.apache.camel.component.jms.JmsProducer.process(
> JmsProducer.java:132)
>         at org.apache.camel.util.AsyncProcessorHelper.process(
> AsyncProcessorHelper.java:73)
>         at org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(
> SendProcessor.java:120)
>         at org.apache.camel.impl.ProducerCache.doInAsyncProducer(
> ProducerCache.java:292)
>         at org.apache.camel.processor.SendProcessor.process(
> SendProcessor.java:115)
>         at org.apache.camel.util.AsyncProcessorHelper.process(
> AsyncProcessorHelper.java:73)
>         at org.apache.camel.processor.DelegateAsyncProcessor.processNext(
> DelegateAsyncProcessor.java:99)
>         at org.apache.camel.processor.DelegateAsyncProcessor.process(
> DelegateAsyncProcessor.java:90)
>         at org.apache.camel.management.InstrumentationProcessor.process(
> InstrumentationProcessor.java:73)
>         at org.apache.camel.util.AsyncProcessorHelper.process(
> AsyncProcessorHelper.java:73)
>         at org.apache.camel.processor.DelegateAsyncProcessor.processNext(
> DelegateAsyncProcessor.java:99)
>         at org.apache.camel.processor.DelegateAsyncProcessor.process(
> DelegateAsyncProcessor.java:90)
>         at org.apache.camel.processor.interceptor.TraceInterceptor.
> process(TraceInterceptor.java:91)
>         at org.apache.camel.util.AsyncProcessorHelper.process(
> AsyncProcessorHelper.java:73)
>         at org.apache.camel.processor.RedeliveryErrorHandler.
> processErrorHandler(RedeliveryErrorHandler.java:330)
>         at org.apache.camel.processor.RedeliveryErrorHandler.process(
> RedeliveryErrorHandler.java:220)
>         at org.apache.camel.processor.RouteContextProcessor.processNext(
> RouteContextProcessor.java:45)
>         at org.apache.camel.processor.DelegateAsyncProcessor.process(
> DelegateAsyncProcessor.java:90)
>         at org.apache.camel.processor.interceptor.DefaultChannel.
> process(DefaultChannel.java:303)
>         at org.apache.camel.util.AsyncProcessorHelper.process(
> AsyncProcessorHelper.java:73)
>         at org.apache.camel.processor.Pipeline.process(Pipeline.java:117)
>         at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
>         at org.apache.camel.processor.RouteContextProcessor.processNext(
> RouteContextProcessor.java:45)
>         at org.apache.camel.processor.DelegateAsyncProcessor.process(
> DelegateAsyncProcessor.java:90)
>         at org.apache.camel.processor.UnitOfWorkProcessor.processAsync(
> UnitOfWorkProcessor.java:150)
>         at org.apache.camel.processor.UnitOfWorkProcessor.process(
> UnitOfWorkProcessor.java:117)
>         at org.apache.camel.processor.RouteInflightRepositoryProcess
> or.processNext(RouteInflightRepositoryProcessor.java:48)
>         at org.apache.camel.processor.DelegateAsyncProcessor.process(
> DelegateAsyncProcessor.java:90)
>         at org.apache.camel.util.AsyncProcessorHelper.process(
> AsyncProcessorHelper.java:73)
>         at org.apache.camel.processor.DelegateAsyncProcessor.processNext(
> DelegateAsyncProcessor.java:99)
>         at org.apache.camel.processor.DelegateAsyncProcessor.process(
> DelegateAsyncProcessor.java:90)
>         at org.apache.camel.management.InstrumentationProcessor.process(
> InstrumentationProcessor.java:73)
>         at org.apache.camel.util.AsyncProcessorHelper.process(
> AsyncProcessorHelper.java:99)
>         at org.apache.camel.processor.DelegateAsyncProcessor.process(
> DelegateAsyncProcessor.java:86)
>         at org.apache.camel.component.timer.TimerConsumer.
> sendTimerExchange(TimerConsumer.java:134)
>         at org.apache.camel.component.timer.TimerConsumer$1.run(
> TimerConsumer.java:63)
>         at java.util.TimerThread.mainLoop(Timer.java:555)
>         at java.util.TimerThread.run(Timer.java:505)
> Caused by: javax.jms.JMSException: Channel was inactive for too (>30000)
> long: tcp://192.168.10.144:61617
>         at org.apache.activemq.util.JMSExceptionSupport.create(
> JMSExceptionSupport.java:49)
>         at org.apache.activemq.ActiveMQConnection.syncSendPacket(
> ActiveMQConnection.java:1377)
>         at org.apache.activemq.ActiveMQConnection.syncSendPacket(
> ActiveMQConnection.java:1305)
>         at org.apache.activemq.ActiveMQSession.send(
> ActiveMQSession.java:1795)
>         at org.apache.activemq.ActiveMQMessageProducer.send(
> ActiveMQMessageProducer.java:277)
>         at org.apache.activemq.ActiveMQMessageProducer.send(
> ActiveMQMessageProducer.java:212)
>         at org.apache.activemq.pool.PooledProducer.send(
> PooledProducer.java:79)
>         at org.apache.activemq.pool.PooledProducer.send(
> PooledProducer.java:62)
>         at org.springframework.jms.core.JmsTemplate.doSend(
> JmsTemplate.java:592)
>         at org.apache.camel.component.jms.JmsConfiguration$
> CamelJmsTemplate.doSend(JmsConfiguration.java:276)
>         at org.apache.camel.component.jms.JmsConfiguration$
> CamelJmsTemplate.doSendToDestination(JmsConfiguration.java:215)
>         at org.apache.camel.component.jms.JmsConfiguration$
> CamelJmsTemplate.access$100(JmsConfiguration.java:157)
>         at org.apache.camel.component.jms.JmsConfiguration$
> CamelJmsTemplate$3.doInJms(JmsConfiguration.java:191)
>         at org.springframework.jms.core.JmsTemplate.execute(
> JmsTemplate.java:466)
>         ... 41 more
> Caused by: org.apache.activemq.transport.InactivityIOException: Channel
> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>         at org.apache.activemq.transport.AbstractInactivityMonitor$4.
> run(AbstractInactivityMonitor.java:209)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> 22 Mar 2015 08:42:39,546 15  WARN timer.TimerConsumer Error processing
> exchange. Exchange[Message: org.apache.uima.ducc.transport.event.
> JdStateDuccEvent@db25b23]. Caused by: [org.springframework.jms.UncategorizedJmsException
> - Uncategorized exception occured during JMS processing; nested exception
> is javax.jms.JMSException: Channel was inactive for too (>30000) long:
> tcp://192.168.10.144:61617]
> org.springframework.jms.UncategorizedJmsException: Uncategorized
> exception occured during JMS processing; nested exception is
> javax.jms.JMSException: Channel was inactive for too (>30000) long: tcp://
> 192.168.10.144:61617
>         at org.springframework.jms.support.JmsUtils.
> convertJmsAccessException(JmsUtils.java:316)
>         at org.springframework.jms.support.JmsAccessor.
> convertJmsAccessException(JmsAccessor.java:168)
>         at org.springframework.jms.core.JmsTemplate.execute(
> JmsTemplate.java:469)
>         at org.apache.camel.component.jms.JmsConfiguration$
> CamelJmsTemplate.send(JmsConfiguration.java:188)
>         at org.apache.camel.component.jms.JmsProducer.doSend(
> JmsProducer.java:398)
>         at org.apache.camel.component.jms.JmsProducer.processInOnly(
> JmsProducer.java:352)
>         at org.apache.camel.component.jms.JmsProducer.process(
> JmsProducer.java:132)
>         at org.apache.camel.util.AsyncProcessorHelper.process(
> AsyncProcessorHelper.java:73)
>         at org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(
> SendProcessor.java:120)
>         at org.apache.camel.impl.ProducerCache.doInAsyncProducer(
> ProducerCache.java:292)
>         at org.apache.camel.processor.SendProcessor.process(
> SendProcessor.java:115)
>         at org.apache.camel.util.AsyncProcessorHelper.process(
> AsyncProcessorHelper.java:73)
>         at org.apache.camel.processor.DelegateAsyncProcessor.processNext(
> DelegateAsyncProcessor.java:99)
>         at org.apache.camel.processor.DelegateAsyncProcessor.process(
> DelegateAsyncProcessor.java:90)
>         at org.apache.camel.management.InstrumentationProcessor.process(
> InstrumentationProcessor.java:73)
>         at org.apache.camel.util.AsyncProcessorHelper.process(
> AsyncProcessorHelper.java:73)
>         at org.apache.camel.processor.DelegateAsyncProcessor.processNext(
> DelegateAsyncProcessor.java:99)
>         at org.apache.camel.processor.DelegateAsyncProcessor.process(
> DelegateAsyncProcessor.java:90)
>         at org.apache.camel.processor.interceptor.TraceInterceptor.
> process(TraceInterceptor.java:91)
>         at org.apache.camel.util.AsyncProcessorHelper.process(
> AsyncProcessorHelper.java:73)
>         at org.apache.camel.processor.RedeliveryErrorHandler.
> processErrorHandler(RedeliveryErrorHandler.java:330)
>         at org.apache.camel.processor.RedeliveryErrorHandler.process(
> RedeliveryErrorHandler.java:220)
>         at org.apache.camel.processor.RouteContextProcessor.processNext(
> RouteContextProcessor.java:45)
>         at org.apache.camel.processor.DelegateAsyncProcessor.process(
> DelegateAsyncProcessor.java:90)
>         at org.apache.camel.processor.interceptor.DefaultChannel.
> process(DefaultChannel.java:303)
>         at org.apache.camel.util.AsyncProcessorHelper.process(
> AsyncProcessorHelper.java:73)
>         at org.apache.camel.processor.Pipeline.process(Pipeline.java:117)
>         at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
>         at org.apache.camel.processor.RouteContextProcessor.processNext(
> RouteContextProcessor.java:45)
>         at org.apache.camel.processor.DelegateAsyncProcessor.process(
> DelegateAsyncProcessor.java:90)
>         at org.apache.camel.processor.UnitOfWorkProcessor.processAsync(
> UnitOfWorkProcessor.java:150)
>         at org.apache.camel.processor.UnitOfWorkProcessor.process(
> UnitOfWorkProcessor.java:117)
>         at org.apache.camel.processor.RouteInflightRepositoryProcess
> or.processNext(RouteInflightRepositoryProcessor.java:48)
>         at org.apache.camel.processor.DelegateAsyncProcessor.process(
> DelegateAsyncProcessor.java:90)
>         at org.apache.camel.util.AsyncProcessorHelper.process(
> AsyncProcessorHelper.java:73)
>         at org.apache.camel.processor.DelegateAsyncProcessor.processNext(
> DelegateAsyncProcessor.java:99)
>         at org.apache.camel.processor.DelegateAsyncProcessor.process(
> DelegateAsyncProcessor.java:90)
>         at org.apache.camel.management.InstrumentationProcessor.process(
> InstrumentationProcessor.java:73)
>         at org.apache.camel.util.AsyncProcessorHelper.process(
> AsyncProcessorHelper.java:99)
>         at org.apache.camel.processor.DelegateAsyncProcessor.process(
> DelegateAsyncProcessor.java:86)
>         at org.apache.camel.component.timer.TimerConsumer.
> sendTimerExchange(TimerConsumer.java:134)
>         at org.apache.camel.component.timer.TimerConsumer$1.run(
> TimerConsumer.java:63)
>         at java.util.TimerThread.mainLoop(Timer.java:555)
>         at java.util.TimerThread.run(Timer.java:505)
> Caused by: javax.jms.JMSException: Channel was inactive for too (>30000)
> long: tcp://192.168.10.144:61617
>         at org.apache.activemq.util.JMSExceptionSupport.create(
> JMSExceptionSupport.java:49)
>         at org.apache.activemq.ActiveMQConnection.syncSendPacket(
> ActiveMQConnection.java:1377)
>         at org.apache.activemq.ActiveMQConnection.syncSendPacket(
> ActiveMQConnection.java:1305)
>         at org.apache.activemq.ActiveMQSession.send(
> ActiveMQSession.java:1795)
>         at org.apache.activemq.ActiveMQMessageProducer.send(
> ActiveMQMessageProducer.java:277)
>         at org.apache.activemq.ActiveMQMessageProducer.send(
> ActiveMQMessageProducer.java:212)
>         at org.apache.activemq.pool.PooledProducer.send(
> PooledProducer.java:79)
>         at org.apache.activemq.pool.PooledProducer.send(
> PooledProducer.java:62)
>         at org.springframework.jms.core.JmsTemplate.doSend(
> JmsTemplate.java:592)
>         at org.apache.camel.component.jms.JmsConfiguration$
> CamelJmsTemplate.doSend(JmsConfiguration.java:276)
>         at org.apache.camel.component.jms.JmsConfiguration$
> CamelJmsTemplate.doSendToDestination(JmsConfiguration.java:215)
>         at org.apache.camel.component.jms.JmsConfiguration$
> CamelJmsTemplate.access$100(JmsConfiguration.java:157)
>         at org.apache.camel.component.jms.JmsConfiguration$
> CamelJmsTemplate$3.doInJms(JmsConfiguration.java:191)
>         at org.springframework.jms.core.JmsTemplate.execute(
> JmsTemplate.java:466)
>         ... 41 more
> Caused by: org.apache.activemq.transport.InactivityIOException: Channel
> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>         at org.apache.activemq.transport.AbstractInactivityMonitor$4.
> run(AbstractInactivityMonitor.java:209)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> 22 Mar 2015 08:43:19,482 14  WARN jms.DefaultJmsMessageListenerContainer
> Setup of JMS message listener invoker failed for destination
> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel
> was inactive for too (>30000) long: tcp://192.168.10.144:61617
> 22 Mar 2015 08:44:59,642 14  WARN jms.DefaultJmsMessageListenerContainer
> Setup of JMS message listener invoker failed for destination
> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel
> was inactive for too (>30000) long: tcp://192.168.10.144:61617
> 22 Mar 2015 08:46:10,729 14  WARN jms.DefaultJmsMessageListenerContainer
> Setup of JMS message listener invoker failed for destination
> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel
> was inactive for too (>30000) long: tcp://192.168.10.144:61617
> 22 Mar 2015 08:48:05,731 14  WARN jms.DefaultJmsMessageListenerContainer
> Setup of JMS message listener invoker failed for destination
> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel
> was inactive for too (>30000) long: tcp://192.168.10.144:61617
> 22 Mar 2015 08:49:49,723 14  WARN jms.DefaultJmsMessageListenerContainer
> Setup of JMS message listener invoker failed for destination
> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel
> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>
>
>
> Please help me to find out root cause of this and how can I kill a job or
> start a new one when any job gets stuck after a certain time..
>
> --
> Thanks and Regards,
> Satya Nand Kanodia
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message