uima-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pierre Pujalon <ppuja...@yahoo.fr>
Subject Re: Lost Connection to Delegate
Date Thu, 31 Mar 2011 13:55:35 GMT
JC

I have a 15000ms process timeout for the UIMAEngine and I have a 15000ms 
timeout for the remoteDelegate

I haven't found how to set a timeout for the aggregate to be honest.

You are right I have this exception in the console
javax.jms.InvalidDestinationException: Cannot publish to a deleted 
Destination: temp-queue://ID:${brokerUrl}-55450-1301578320924-0:0:1
         at 
org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1696)
         at 
org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:231)
         at 
org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:300)
         at 
org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.send(JmsEndpointConnection_impl.java:574)
         at 
org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.send(JmsEndpointConnection_impl.java:507)
         at 
org.apache.uima.adapter.jms.activemq.JmsOutputChannel.dispatch(JmsOutputChannel.java:1366)
         at 
org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendCasToRemoteEndpoint(JmsOutputChannel.java:1527)
         at 
org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendReply(JmsOutputChannel.java:689)
         at 
org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.sendReplyToRemoteClient(AggregateAnalysisEngineController_impl.java:2033)
         at 
org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.replyToClient(AggregateAnalysisEngineController_impl.java:2170)
         at 
org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.finalStep(AggregateAnalysisEngineController_impl.java:1734)
         at 
org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.executeFlowStep(AggregateAnalysisEngineController_impl.java:2317)
         at 
org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.process(AggregateAnalysisEngineController_impl.java:1229)
         at 
org.apache.uima.aae.handler.HandlerBase.invokeProcess(HandlerBase.java:118)
         at 
org.apache.uima.aae.handler.input.ProcessResponseHandler.cancelTimerAndProcess(ProcessResponseHandler.java:108)
         at 
org.apache.uima.aae.handler.input.ProcessResponseHandler.handleProcessResponseWithCASReference(ProcessResponseHandler.java:387)
         at 
org.apache.uima.aae.handler.input.ProcessResponseHandler.handle(ProcessResponseHandler.java:656)
         at 
org.apache.uima.aae.handler.HandlerBase.delegate(HandlerBase.java:149)
         at 
org.apache.uima.aae.handler.input.ProcessRequestHandler_impl.handle(ProcessRequestHandler_impl.java:973)
         at 
org.apache.uima.aae.spi.transport.vm.UimaVmMessageListener.onMessage(UimaVmMessageListener.java:107)
         at 
org.apache.uima.aae.spi.transport.vm.UimaVmMessageDispatcher$1.run(UimaVmMessageDispatcher.java:70)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
         at java.lang.Thread.run(Thread.java:619)


Thanks
Pierre

Le 31/03/2011 15:43, Jaroslaw Cwiklik a écrit :
> Pierre, since the aggregate received the three CASes and these are never
> returned to the application client I suspect that a temp reply queue
> associated with the three CASes is no longer valid. Check your aggregate log
> for exceptions. Perhaps the aggregate is trying to return the CASes but it
> fails on send since the temp reply queue no longer exists. Just to be clear
> do you define Process timeout in your application client? It wasnt clear to
> me if the timeout of 15000ms is in the aggregate client or the application
> client.
>
> JC
>
> On Thu, Mar 31, 2011 at 9:24 AM, Pierre Pujalon<ppujalon@yahoo.fr>  wrote:
>
>> JC,
>>
>> I had to work on this topic and add instrumentation a bit to try to have a
>> clearer vision. The basic idea is to test robustness of the processing when
>> the ActiveMQ have a downtime (generally only for milliseconds). This
>> scenario is a real life scenario.
>>
>> I have an application client using the aggregate client with
>> UimaAsynchronousEngine. I have a pool of 4 CASes, a  timeout set to 15000ms
>> in the client and not CASMultiplier in the aggregate.
>>
>> When JMS gets down and up. I receive in my CallbackListener only one
>> exception out of the four CASes (the 15000ms timeout) . The three others
>> never trigger any timeout.
>> Actually, it seems that thoses CASes are being processed in the aggregate
>> as I have the trace of the annotators in the aggregate log. But they never
>> go back to the reply queue.
>>
>> I traced the CASId that the engine sends and receives in the listener.
>> Let's say I have id1, Id2, id3 and id4.
>>
>> As the JMS gets down and up,  one of the case (id1) is released . The
>> application client send another CAS . As this very point, the method sendCas
>> returns a CASId id5 but in the CallbackListener the onBeforeMessageSend
>> status tells me the CASid is id4.  It is confirmed with the aggregate which
>> logs the message :
>> Duplicate Request With Cas Reference Id: (id4) Received. Ignoring
>> Duplicate.
>>
>> The CasPool in the UIMAEngines is full and the client stalls.
>> Restarting the UIMAAsynchronousEngine is the only solution to continue
>> processing, the ag
>> My guess is that it comes from the UIMAAsynchronousEngine.
>> I hope it is clear
>>
>> Pierre
>> PS : I set the timeout to 5 seconds, and I have a total different scenario
>> which is no better :)
>>
>>
>>
>> Le 29/03/2011 18:56, Jaroslaw Cwiklik a écrit :
>>
>>   Pierre, do you declare a timeout on Process in your aggregate deployment
>>> descriptor for the remote delegate? If not, perhaps CASes were sent to the
>>> remote delegate and subsequently dropped because the delegate was not
>>> unable
>>> to deliver them to the aggregate client. A timeout would help to recover
>>> these CASes.
>>>
>>> Do you know where you are running out of CASes? Is it in application
>>> client
>>> or aggregate client? Use jConsole to attach to each jvm and examine the
>>> threads. Do you use a Cas Multiplier in the aggregate? How big is its
>>> pool?
>>>
>>> -JC
>>>
>>> On Tue, Mar 29, 2011 at 12:03 PM, Pierre Pujalon<ppujalon@yahoo.fr>
>>>   wrote:
>>>
>>>   JC,
>>>> I tested the work around  and a listener to the delegate is indeed
>>>> recreated. The CAS that were being processed pass through all annotators.
>>>> But now I have other issues, but it is not the same each time I tried
>>>> .Either the client is losing connection, or get block on the getCAS
>>>> method
>>>> as if cas process during the JMS down time were not release or the
>>>> aggregate
>>>> or the aggregate client get stalled...
>>>> I need to dig a little bit into this because it does not make many sense
>>>> right now...
>>>>
>>>> What bother me is that if I don't make the annotator delegated, the
>>>> aggregate client  always recover clean with this scenario.
>>>>
>>>> Anyway Thanks
>>>> Pierre
>>>>
>>>>
>>>>
>>>> Le 28/03/2011 18:07, Jaroslaw Cwiklik a écrit :
>>>>
>>>>   Pierre, in my last posting I meant maxRetries=1, not retryCount=1.
>>>>
>>>>> JC
>>>>>
>>>>> On Mon, Mar 28, 2011 at 12:05 PM, Jaroslaw Cwiklik<uimaee@gmail.com>
>>>>>   wrote:
>>>>>
>>>>>   Pierre, I've identified the bug in UIMA AS code that causes the problem
>>>>>
>>>>>> described. As a workaround, set the retryCount=1 in the error handling
>>>>>> configuration for the delegate defined in your deployment descriptor.
>>>>>> Here
>>>>>> is an example:
>>>>>>
>>>>>> <remoteAnalysisEngine key="NoOp">
>>>>>>          <inputQueue endpoint="NoOpAnnotatorQueue"
>>>>>> brokerURL="${BrokerURL}"/>
>>>>>>          <serializer method="xmi"/>
>>>>>>          <asyncAggregateErrorConfiguration>
>>>>>>                              <processCasErrors maxRetries="1"
>>>>>> timeout="15000" thresholdCount="1" thresholdAction="terminate" />
>>>>>>           </asyncAggregateErrorConfiguration>
>>>>>> </remoteAnalysisEngine>
>>>>>>
>>>>>> This should force creation of a new listener on the delegate reply
>>>>>> queue.
>>>>>>
>>>>>> The listener recovery should be done with no retry being set and
I will
>>>>>> make necessary changes in the code to support that use case.
>>>>>>
>>>>>> Thanks for finding the bug!
>>>>>>
>>>>>> JC
>>>>>>
>>>>>> On Tue, Mar 22, 2011 at 1:12 PM, Jaroslaw Cwiklik<uimaee@gmail.com
>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>   Pierre, this appears to be a bug. In the scenario you've described
the
>>>>>>
>>>>>>> uima as aggregate client should detect a lost connection (which
it
>>>>>>> does)
>>>>>>> and
>>>>>>> create a new one on a subsequent process CAS request destined
to the
>>>>>>> remote
>>>>>>> delegate. I will attempt to re-create your scenario and see if
this is
>>>>>>> in
>>>>>>> fact a bug.
>>>>>>>
>>>>>>> Thanks, JC
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Mar 22, 2011 at 7:11 AM, Pierre Pujalon<ppujalon@yahoo.fr
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>    Hi,
>>>>>>>
>>>>>>>>   I'm currently using UIMA-AS 2.3.1rc5 and I am having issues
with
>>>>>>>> asynchronous delegated ..
>>>>>>>>
>>>>>>>> I have a chain of several annotators. One of them is an asynchronous
>>>>>>>> delegate as it is very resource consuming.
>>>>>>>> So I have one UIMA service with a simpleChainDescriptor and
several
>>>>>>>> UIMA
>>>>>>>> services with myDelegateAnnotatorDescriptor.
>>>>>>>>
>>>>>>>>   When the activeMQ broker fails and restart, all my UIMA
services
>>>>>>>> resfresh succesfully their JMS connection. But it appears
that the
>>>>>>>> service
>>>>>>>> holding the processing chain (simpleChainDescriptor)  itself
does not
>>>>>>>> refresh the JMS connection to the delegate JMS queue.
>>>>>>>>   When a new CAS is processed, I have the following exception
in the
>>>>>>>> console !!!! and the processing of the cas stop silently
and passes
>>>>>>>> to
>>>>>>>> the
>>>>>>>> next cas.
>>>>>>>>   If I restart the service holding the chain, everything
go back to
>>>>>>>> normal.
>>>>>>>>
>>>>>>>>   Is this a bug or am I missing something?
>>>>>>>>
>>>>>>>>   Thanks,
>>>>>>>>   Pierre
>>>>>>>>
>>>>>>>>
>>>>>>>> ATTENTION: Uima AS Service:simpleChainDescriptor Listener
Established
>>>>>>>> Connection to Broker: tcp://brokerurl:61616
>>>>>>>> 22 mars 2011 11:11:00
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer
>>>>>>>> handleListenerSetupFailure
>>>>>>>> ATTENTION: Uima AS Service:simpleChainDescriptor Listener
Established
>>>>>>>> Connection to Broker: tcp://brokerurl:61616
>>>>>>>> 22 mars 2011 11:11:00
>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel
>>>>>>>> invalidateConnectionAndEndpoints
>>>>>>>> INFO: Controller simpleChainDescriptor Invalidating JMS Connection
To
>>>>>>>> Broker tcp://brokerurl:61616 and Closing Sessions To Delegates
>>>>>>>> 22 mars 2011 11:11:01
>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl
send
>>>>>>>> ATTENTION: Service: simpleChainDescriptor Runtime Exception
>>>>>>>> 22 mars 2011 11:11:01
>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl
send
>>>>>>>> ATTENTION:
>>>>>>>> org.apache.uima.aae.error.DelegateConnectionLostException:
>>>>>>>> Controller:simpleChainDescriptor Lost Connection to
>>>>>>>> Delegate:myDelegateAnnotatorDescriptor
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.send(JmsEndpointConnection_impl.java:536)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.send(JmsEndpointConnection_impl.java:507)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.dispatch(JmsOutputChannel.java:1366)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendCasToRemoteEndpoint(JmsOutputChannel.java:1527)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.serializeCasAndSend(JmsOutputChannel.java:658)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendRequest(JmsOutputChannel.java:610)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.dispatch(AggregateAnalysisEngineController_impl.java:2381)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.dispatchProcessRequest(AggregateAnalysisEngineController_impl.java:2421)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.simpleStep(AggregateAnalysisEngineController_impl.java:1294)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.executeFlowStep(AggregateAnalysisEngineController_impl.java:2302)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.process(AggregateAnalysisEngineController_impl.java:1229)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.aae.handler.HandlerBase.invokeProcess(HandlerBase.java:118)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.cancelTimerAndProcess(ProcessResponseHandler.java:108)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.handleProcessResponseWithCASReference(ProcessResponseHandler.java:387)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.handle(ProcessResponseHandler.java:656)
>>>>>>>>         at
>>>>>>>>
>>>>>>>> org.apache.uima.aae.handler.HandlerBase.delegate(HandlerBase.java:149)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.aae.handler.input.ProcessRequestHandler_impl.handle(ProcessRequestHandler_impl.java:973)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.aae.spi.transport.vm.UimaVmMessageListener.onMessage(UimaVmMessageListener.java:107)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.aae.spi.transport.vm.UimaVmMessageDispatcher$1.run(UimaVmMessageDispatcher.java:70)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>         at java.lang.Thread.run(Thread.java:619)
>>>>>>>> 22 mars 2011 11:11:01
>>>>>>>> org.apache.uima.aae.error.handler.ProcessCasErrorHandler
handleError
>>>>>>>> ATTENTION: Service: simpleChainDescriptor Runtime Exception
>>>>>>>> 22 mars 2011 11:11:01
>>>>>>>> org.apache.uima.aae.error.handler.ProcessCasErrorHandler
handleError
>>>>>>>> ATTENTION:
>>>>>>>> org.apache.uima.aae.error.DelegateConnectionLostException:
>>>>>>>> Controller:simpleChainDescriptor Lost Connection to
>>>>>>>> Delegate:myDelegateAnnotatorDescriptor
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.send(JmsEndpointConnection_impl.java:536)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.send(JmsEndpointConnection_impl.java:507)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.dispatch(JmsOutputChannel.java:1366)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendCasToRemoteEndpoint(JmsOutputChannel.java:1527)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.serializeCasAndSend(JmsOutputChannel.java:658)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendRequest(JmsOutputChannel.java:610)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.dispatch(AggregateAnalysisEngineController_impl.java:2381)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.dispatchProcessRequest(AggregateAnalysisEngineController_impl.java:2421)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.simpleStep(AggregateAnalysisEngineController_impl.java:1294)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.executeFlowStep(AggregateAnalysisEngineController_impl.java:2302)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.process(AggregateAnalysisEngineController_impl.java:1229)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.aae.handler.HandlerBase.invokeProcess(HandlerBase.java:118)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.cancelTimerAndProcess(ProcessResponseHandler.java:108)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.handleProcessResponseWithCASReference(ProcessResponseHandler.java:387)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.handle(ProcessResponseHandler.java:656)
>>>>>>>>         at
>>>>>>>>
>>>>>>>> org.apache.uima.aae.handler.HandlerBase.delegate(HandlerBase.java:149)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.aae.handler.input.ProcessRequestHandler_impl.handle(ProcessRequestHandler_impl.java:973)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.aae.spi.transport.vm.UimaVmMessageListener.onMessage(UimaVmMessageListener.java:107)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.uima.aae.spi.transport.vm.UimaVmMessageDispatcher$1.run(UimaVmMessageDispatcher.java:70)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>         at
>>>>>>>>
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>         at java.lang.Thread.run(Thread.java:619)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>

Mime
View raw message