Return-Path: Delivered-To: apmail-uima-user-archive@www.apache.org Received: (qmail 28324 invoked from network); 31 Mar 2011 13:44:26 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 31 Mar 2011 13:44:26 -0000 Received: (qmail 69002 invoked by uid 500); 31 Mar 2011 13:44:24 -0000 Delivered-To: apmail-uima-user-archive@uima.apache.org Received: (qmail 68958 invoked by uid 500); 31 Mar 2011 13:44:24 -0000 Mailing-List: contact user-help@uima.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@uima.apache.org Delivered-To: mailing list user@uima.apache.org Received: (qmail 68919 invoked by uid 99); 31 Mar 2011 13:44:24 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 13:44:24 +0000 X-ASF-Spam-Status: No, hits=4.0 required=5.0 tests=FREEMAIL_FROM,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of uimaee@gmail.com designates 209.85.161.47 as permitted sender) Received: from [209.85.161.47] (HELO mail-fx0-f47.google.com) (209.85.161.47) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 13:44:17 +0000 Received: by fxm19 with SMTP id 19so2086153fxm.6 for ; Thu, 31 Mar 2011 06:43:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=DWyzHf4BFwBikf1xtIRcF+bnCEqzwX0R1p99RD8A5Ks=; b=HHu27sp5s3qxVazuHsc67Z+AfeDlBCZ3wX/13iGv7B+q7zJLnO3WSgVr2X2oWbDseY H+/v/ZdldK/oR3c1vh8c3vPjnJBqij4NiOjqGe13fjDycx8busXjjNt1EBl0eSdIm508 uK/44MLTjfVNvCfHLf5uEPv9IuLLCyQqIbTwM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=vVqKxrzc6SPYPZp22+94bCxF45Vr7H3uXURiYfVGSl/6LPq3Q8cnUqqxSZ+ELll3RH YTXnOAfIcGh4G1rYBUq3Pekd/3CkYtXQeeZhRzEV7uYFJFj0h9YVmsktchRKH8c2SjUz JS/gYFsbY17IfXig1J8qnpgEnBsjhpgEN/jm8= MIME-Version: 1.0 Received: by 10.223.160.8 with SMTP id l8mr2462080fax.114.1301579037104; Thu, 31 Mar 2011 06:43:57 -0700 (PDT) Received: by 10.223.105.17 with HTTP; Thu, 31 Mar 2011 06:43:56 -0700 (PDT) In-Reply-To: <4D948099.7030304@yahoo.fr> References: <4D8883F9.2010003@yahoo.fr> <4D9202C5.5040202@yahoo.fr> <4D948099.7030304@yahoo.fr> Date: Thu, 31 Mar 2011 09:43:56 -0400 Message-ID: Subject: Re: Lost Connection to Delegate From: Jaroslaw Cwiklik To: user@uima.apache.org Content-Type: multipart/alternative; boundary=0023542750002762f3049fc77c49 X-Virus-Checked: Checked by ClamAV on apache.org --0023542750002762f3049fc77c49 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Pierre, since the aggregate received the three CASes and these are never returned to the application client I suspect that a temp reply queue associated with the three CASes is no longer valid. Check your aggregate lo= g for exceptions. Perhaps the aggregate is trying to return the CASes but it fails on send since the temp reply queue no longer exists. Just to be clear do you define Process timeout in your application client? It wasnt clear to me if the timeout of 15000ms is in the aggregate client or the application client. JC On Thu, Mar 31, 2011 at 9:24 AM, Pierre Pujalon wrote: > JC, > > I had to work on this topic and add instrumentation a bit to try to have = a > clearer vision. The basic idea is to test robustness of the processing wh= en > the ActiveMQ have a downtime (generally only for milliseconds). This > scenario is a real life scenario. > > I have an application client using the aggregate client with > UimaAsynchronousEngine. I have a pool of 4 CASes, a timeout set to 15000= ms > in the client and not CASMultiplier in the aggregate. > > When JMS gets down and up. I receive in my CallbackListener only one > exception out of the four CASes (the 15000ms timeout) . The three others > never trigger any timeout. > Actually, it seems that thoses CASes are being processed in the aggregate > as I have the trace of the annotators in the aggregate log. But they neve= r > go back to the reply queue. > > I traced the CASId that the engine sends and receives in the listener. > Let's say I have id1, Id2, id3 and id4. > > As the JMS gets down and up, one of the case (id1) is released . The > application client send another CAS . As this very point, the method send= Cas > returns a CASId id5 but in the CallbackListener the onBeforeMessageSend > status tells me the CASid is id4. It is confirmed with the aggregate whi= ch > logs the message : > Duplicate Request With Cas Reference Id: (id4) Received. Ignoring > Duplicate. > > The CasPool in the UIMAEngines is full and the client stalls. > Restarting the UIMAAsynchronousEngine is the only solution to continue > processing, the ag > My guess is that it comes from the UIMAAsynchronousEngine. > I hope it is clear > > Pierre > PS : I set the timeout to 5 seconds, and I have a total different scenari= o > which is no better :) > > > > Le 29/03/2011 18:56, Jaroslaw Cwiklik a =E9crit : > > Pierre, do you declare a timeout on Process in your aggregate deployment >> descriptor for the remote delegate? If not, perhaps CASes were sent to t= he >> remote delegate and subsequently dropped because the delegate was not >> unable >> to deliver them to the aggregate client. A timeout would help to recover >> these CASes. >> >> Do you know where you are running out of CASes? Is it in application >> client >> or aggregate client? Use jConsole to attach to each jvm and examine the >> threads. Do you use a Cas Multiplier in the aggregate? How big is its >> pool? >> >> -JC >> >> On Tue, Mar 29, 2011 at 12:03 PM, Pierre Pujalon >> wrote: >> >> JC, >>> I tested the work around and a listener to the delegate is indeed >>> recreated. The CAS that were being processed pass through all annotator= s. >>> But now I have other issues, but it is not the same each time I tried >>> .Either the client is losing connection, or get block on the getCAS >>> method >>> as if cas process during the JMS down time were not release or the >>> aggregate >>> or the aggregate client get stalled... >>> I need to dig a little bit into this because it does not make many sens= e >>> right now... >>> >>> What bother me is that if I don't make the annotator delegated, the >>> aggregate client always recover clean with this scenario. >>> >>> Anyway Thanks >>> Pierre >>> >>> >>> >>> Le 28/03/2011 18:07, Jaroslaw Cwiklik a =E9crit : >>> >>> Pierre, in my last posting I meant maxRetries=3D1, not retryCount=3D1. >>> >>>> JC >>>> >>>> On Mon, Mar 28, 2011 at 12:05 PM, Jaroslaw Cwiklik >>>> wrote: >>>> >>>> Pierre, I've identified the bug in UIMA AS code that causes the probl= em >>>> >>>>> described. As a workaround, set the retryCount=3D1 in the error handl= ing >>>>> configuration for the delegate defined in your deployment descriptor. >>>>> Here >>>>> is an example: >>>>> >>>>> >>>>> >>>> brokerURL=3D"${BrokerURL}"/> >>>>> >>>>> >>>>> >>>> timeout=3D"15000" thresholdCount=3D"1" thresholdAction=3D"terminate" = /> >>>>> >>>>> >>>>> >>>>> This should force creation of a new listener on the delegate reply >>>>> queue. >>>>> >>>>> The listener recovery should be done with no retry being set and I wi= ll >>>>> make necessary changes in the code to support that use case. >>>>> >>>>> Thanks for finding the bug! >>>>> >>>>> JC >>>>> >>>>> On Tue, Mar 22, 2011 at 1:12 PM, Jaroslaw Cwiklik>>>> >>>>>> wrote: >>>>>> >>>>> Pierre, this appears to be a bug. In the scenario you've described t= he >>>>> >>>>>> uima as aggregate client should detect a lost connection (which it >>>>>> does) >>>>>> and >>>>>> create a new one on a subsequent process CAS request destined to the >>>>>> remote >>>>>> delegate. I will attempt to re-create your scenario and see if this = is >>>>>> in >>>>>> fact a bug. >>>>>> >>>>>> Thanks, JC >>>>>> >>>>>> >>>>>> On Tue, Mar 22, 2011 at 7:11 AM, Pierre Pujalon>>>>> >>>>>>> wrote: >>>>>>> >>>>>> Hi, >>>>>> >>>>>>> I'm currently using UIMA-AS 2.3.1rc5 and I am having issues with >>>>>>> asynchronous delegated .. >>>>>>> >>>>>>> I have a chain of several annotators. One of them is an asynchronou= s >>>>>>> delegate as it is very resource consuming. >>>>>>> So I have one UIMA service with a simpleChainDescriptor and several >>>>>>> UIMA >>>>>>> services with myDelegateAnnotatorDescriptor. >>>>>>> >>>>>>> When the activeMQ broker fails and restart, all my UIMA services >>>>>>> resfresh succesfully their JMS connection. But it appears that the >>>>>>> service >>>>>>> holding the processing chain (simpleChainDescriptor) itself does n= ot >>>>>>> refresh the JMS connection to the delegate JMS queue. >>>>>>> When a new CAS is processed, I have the following exception in the >>>>>>> console !!!! and the processing of the cas stop silently and passes >>>>>>> to >>>>>>> the >>>>>>> next cas. >>>>>>> If I restart the service holding the chain, everything go back to >>>>>>> normal. >>>>>>> >>>>>>> Is this a bug or am I missing something? >>>>>>> >>>>>>> Thanks, >>>>>>> Pierre >>>>>>> >>>>>>> >>>>>>> ATTENTION: Uima AS Service:simpleChainDescriptor Listener Establish= ed >>>>>>> Connection to Broker: tcp://brokerurl:61616 >>>>>>> 22 mars 2011 11:11:00 >>>>>>> >>>>>>> >>>>>>> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerCont= ainer >>>>>>> handleListenerSetupFailure >>>>>>> ATTENTION: Uima AS Service:simpleChainDescriptor Listener Establish= ed >>>>>>> Connection to Broker: tcp://brokerurl:61616 >>>>>>> 22 mars 2011 11:11:00 >>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel >>>>>>> invalidateConnectionAndEndpoints >>>>>>> INFO: Controller simpleChainDescriptor Invalidating JMS Connection = To >>>>>>> Broker tcp://brokerurl:61616 and Closing Sessions To Delegates >>>>>>> 22 mars 2011 11:11:01 >>>>>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl sen= d >>>>>>> ATTENTION: Service: simpleChainDescriptor Runtime Exception >>>>>>> 22 mars 2011 11:11:01 >>>>>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl sen= d >>>>>>> ATTENTION: >>>>>>> org.apache.uima.aae.error.DelegateConnectionLostException: >>>>>>> Controller:simpleChainDescriptor Lost Connection to >>>>>>> Delegate:myDelegateAnnotatorDescriptor >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.sen= d(JmsEndpointConnection_impl.java:536) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.sen= d(JmsEndpointConnection_impl.java:507) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.dispatch(JmsO= utputChannel.java:1366) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendCasToRemo= teEndpoint(JmsOutputChannel.java:1527) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.serializeCasA= ndSend(JmsOutputChannel.java:658) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendRequest(J= msOutputChannel.java:610) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_im= pl.dispatch(AggregateAnalysisEngineController_impl.java:2381) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_im= pl.dispatchProcessRequest(AggregateAnalysisEngineController_impl.java:2421) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_im= pl.simpleStep(AggregateAnalysisEngineController_impl.java:1294) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_im= pl.executeFlowStep(AggregateAnalysisEngineController_impl.java:2302) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_im= pl.process(AggregateAnalysisEngineController_impl.java:1229) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.aae.handler.HandlerBase.invokeProcess(HandlerBase.j= ava:118) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.cancelTime= rAndProcess(ProcessResponseHandler.java:108) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.handleProc= essResponseWithCASReference(ProcessResponseHandler.java:387) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.handle(Pro= cessResponseHandler.java:656) >>>>>>> at >>>>>>> >>>>>>> org.apache.uima.aae.handler.HandlerBase.delegate(HandlerBase.java:1= 49) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.aae.handler.input.ProcessRequestHandler_impl.handle= (ProcessRequestHandler_impl.java:973) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.aae.spi.transport.vm.UimaVmMessageListener.onMessag= e(UimaVmMessageListener.java:107) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.aae.spi.transport.vm.UimaVmMessageDispatcher$1.run(= UimaVmMessageDispatcher.java:70) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolEx= ecutor.java:886) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:908) >>>>>>> at java.lang.Thread.run(Thread.java:619) >>>>>>> 22 mars 2011 11:11:01 >>>>>>> org.apache.uima.aae.error.handler.ProcessCasErrorHandler handleErro= r >>>>>>> ATTENTION: Service: simpleChainDescriptor Runtime Exception >>>>>>> 22 mars 2011 11:11:01 >>>>>>> org.apache.uima.aae.error.handler.ProcessCasErrorHandler handleErro= r >>>>>>> ATTENTION: >>>>>>> org.apache.uima.aae.error.DelegateConnectionLostException: >>>>>>> Controller:simpleChainDescriptor Lost Connection to >>>>>>> Delegate:myDelegateAnnotatorDescriptor >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.sen= d(JmsEndpointConnection_impl.java:536) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.sen= d(JmsEndpointConnection_impl.java:507) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.dispatch(JmsO= utputChannel.java:1366) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendCasToRemo= teEndpoint(JmsOutputChannel.java:1527) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.serializeCasA= ndSend(JmsOutputChannel.java:658) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendRequest(J= msOutputChannel.java:610) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_im= pl.dispatch(AggregateAnalysisEngineController_impl.java:2381) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_im= pl.dispatchProcessRequest(AggregateAnalysisEngineController_impl.java:2421) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_im= pl.simpleStep(AggregateAnalysisEngineController_impl.java:1294) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_im= pl.executeFlowStep(AggregateAnalysisEngineController_impl.java:2302) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_im= pl.process(AggregateAnalysisEngineController_impl.java:1229) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.aae.handler.HandlerBase.invokeProcess(HandlerBase.j= ava:118) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.cancelTime= rAndProcess(ProcessResponseHandler.java:108) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.handleProc= essResponseWithCASReference(ProcessResponseHandler.java:387) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.handle(Pro= cessResponseHandler.java:656) >>>>>>> at >>>>>>> >>>>>>> org.apache.uima.aae.handler.HandlerBase.delegate(HandlerBase.java:1= 49) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.aae.handler.input.ProcessRequestHandler_impl.handle= (ProcessRequestHandler_impl.java:973) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.aae.spi.transport.vm.UimaVmMessageListener.onMessag= e(UimaVmMessageListener.java:107) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.uima.aae.spi.transport.vm.UimaVmMessageDispatcher$1.run(= UimaVmMessageDispatcher.java:70) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolEx= ecutor.java:886) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:908) >>>>>>> at java.lang.Thread.run(Thread.java:619) >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> --0023542750002762f3049fc77c49--