Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id A543D200B95 for ; Tue, 27 Sep 2016 15:07:34 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id A3C00160AD2; Tue, 27 Sep 2016 13:07:34 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 8EAA6160AD3 for ; Tue, 27 Sep 2016 15:07:33 +0200 (CEST) Received: (qmail 42581 invoked by uid 500); 27 Sep 2016 13:07:27 -0000 Mailing-List: contact user-help@uima.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@uima.apache.org Delivered-To: mailing list user@uima.apache.org Received: (qmail 42565 invoked by uid 99); 27 Sep 2016 13:07:27 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Sep 2016 13:07:27 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id C38921A54B0 for ; Tue, 27 Sep 2016 13:07:26 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.129 X-Spam-Level: * X-Spam-Status: No, score=1.129 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_REPLY=1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id pvTsRipvz3be for ; Tue, 27 Sep 2016 13:07:22 +0000 (UTC) Received: from mail-vk0-f50.google.com (mail-vk0-f50.google.com [209.85.213.50]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with ESMTPS id 38A225FC22 for ; Tue, 27 Sep 2016 13:07:22 +0000 (UTC) Received: by mail-vk0-f50.google.com with SMTP id z126so12455077vkd.0 for ; Tue, 27 Sep 2016 06:07:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=rLZZIzqMmAnHaAlZLLxHjsFwIj2jb7JDPBvSgDYY44Y=; b=vacYaED5ZFXG6yAmmh5zbJhwTGiOu/eQcHlr8X1LF2GizKf+5Utwouh60Hwn3fr6BM mXXaqINiHlQXEGJFsmOVSqujXRDwCYDHxyn7zGWWMuCwPQD+PUlyO64iGhB/5R0btUPy /rVhmRKxzRlvzvE0kt/9Fuqmpc68TOtuJSYuMVyU6a+2moRXsyvdd7O3Gp5BW4WkeuWZ dWtBpew8qspKzfr4P6tslh6XrOWvjhbPeAWOtjo0dYn3BkCo30JsOdlxxkPsKBucTAo/ ni+i/3VZmKgzbAPhBwzr2gNmkmyIOgX7uvsVCWQ/KCtXiDEd0L0nrn4+0pfjlS9/dNs2 +hrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=rLZZIzqMmAnHaAlZLLxHjsFwIj2jb7JDPBvSgDYY44Y=; b=joN8E7OFXcdyWKoyykyw/54t1kOxHaQtOO93Lt3GtDtojkf94HIgo1Sox9c9TtspTn DuUKRWE9Hou/gNG5G0XlfbUAMB3mja+5ArRl/vnsWwmi0PgpXX3AzyEqcxuEGjmDErQh YKQV8nGin1pEFyVyCb2nN1SUrTXqKgbO/5zb4kMZYxTcUypeLuzqkJQXIF3rZJ+GnrTO eecTpuLkiw5MdjOMLFALflUjw9ShW56WXBOFdcGQllm/A+7wUKrBftppzW0V8WTcB5VI e98E2y2omav/M2FVZAOgJDLDqNf2EvQ/VNraVaVqnu47mRXHtq86L+W/pSb7r+Ci1esh 9tjA== X-Gm-Message-State: AA6/9RmYWLB3kPvtzLPmZ8i5a/jzLYhE7o++O58ixYIdCyo+xPc6aodsl2Ryp5fo2xE1LdnpI0muzP8w66hRWw== X-Received: by 10.31.21.80 with SMTP id 77mr13291742vkv.111.1474981636116; Tue, 27 Sep 2016 06:07:16 -0700 (PDT) MIME-Version: 1.0 Received: by 10.31.218.7 with HTTP; Tue, 27 Sep 2016 06:07:15 -0700 (PDT) In-Reply-To: References: From: nelson rivera Date: Tue, 27 Sep 2016 09:07:15 -0400 Message-ID: Subject: Re: Broken conections after ACTIVEMQ restart To: user@uima.apache.org Content-Type: text/plain; charset=UTF-8 archived-at: Tue, 27 Sep 2016 13:07:34 -0000 I was testing with AMQ broker 5.14.0 and i get the same error. not appears to be broker issues, appears to be uima-as issues. Any other idea, i am stagnant at work 2016-09-22 13:04 GMT-04:00, Jaroslaw Cwiklik : > Can you install AMQ broker 5.14.0 to see if this is related to broker > issues. > > -jerry > > On Thu, Sep 22, 2016 at 11:29 AM, nelson rivera > wrote: > >> just like that, the parent CAS comes back but the child CAS doesnt. >> >> And the child CAS if processed, because the uima-as log shows the >> lines of the other annotators primitive of the flow processing the >> child CAS, but not comes back to UIMA-AS client. >> >> uima-as log only shows >> "Jms Listener Failed. Endpoint: >> temp-queue://ID:localhsot-H81-M1-33610-1474481197190-1:1:1 Managed By: >> tcp://localhost:61616 Reason: javax.jms.JMSException: >> java.io.EOFException" ........ >> ADVERTENCIA: Jms Listener Failed. Endpoint: XClusterAnalyzerAggregate >> Managed By: tcp://localhost:61616 Reason: >> javax.jms.IllegalStateException: The Consumer is closed.............. >> >> The CAS is send to service after restart the Broker, and how say >> parent CAS comes back but the child CAS doesnt. And if i try quiesce >> the Aggregate Service console shows : "Quiescing UIMA-AS Service. >> Remaining Number of CASes to Process:1". I suppose that is the child >> CAS. >> >> the client if can send more CASes to the service, but not receive >> response, for each CAS send after the one CAS that was send after >> restart broker (for which parent CAS comes back but the child CAS >> doesnt) not comes back parent CAS or child CAS. >> >> More specific: >> scenario 1: >> 1- Start Broker and UIMA-AS aggregate service. Everything is fine. >> >> 2- Restart Broker (without restart the service). UIMA-AS log show : >> "Jms Listener Failed." .... >> 3- Client sends a CAS to UIMA-AS aggregate service.(The parent CAS >> comes back but the child CAS doesnt, to the uima-as client.) >> If i try quiesce the Aggregate Service console shows : "Quiescing >> UIMA-AS Service. Remaining Number of CASes to Process:1" >> >> scenario 2: >> 1- Start Broker and UIMA-AS aggregate service. Everything is fine. >> >> 2- Restart Broker (without restart the service). UIMA-AS log show : >> "Jms Listener Failed." .... >> 3- Client sends a CAS to UIMA-AS aggregate service.(The parent CAS >> comes back but the child CAS doesnt, to the uima-as client) >> 4- Client sends another CAS to UIMA-AS aggregate service.(not comes >> back parent CAS or child CAS , to the uima-as client) >> 5- Client sends another CAS to UIMA-AS aggregate service.(not comes >> back parent CAS or child CAS , to the uima-as client) >> If i try quiesce the Aggregate Service console shows: "Quiescing >> UIMA-AS Service. Remaining Number of CASes to Process:5" >> >> >> >> >> >> 2016-09-22 10:50 GMT-04:00, Jaroslaw Cwiklik : >> > When a parent CAS comes back (after broker is restored), can the client >> > send more CASes to the service and receive responses? I am trying to >> > determine what the real problem is here. >> > >> > 1) JMS Listener not recovering connection >> > 2) UIMA-AS is not handling CAS reply failures when broker connection is >> > stale >> > >> > -jerry >> > >> > On Thu, Sep 22, 2016 at 10:38 AM, Jaroslaw Cwiklik >> > wrote: >> > >> >> Strange. You say that the parent CAS comes back but the child CAS >> doesnt. >> >> Are there any exceptions in the service log? Perhaps the service is >> >> unable >> >> to send the child CAS due to missing broker. There should be something >> in >> >> the uima-as log about this. >> >> >> >> -jerry >> >> >> >> On Thu, Sep 22, 2016 at 10:19 AM, nelson rivera >> >> >> >> wrote: >> >> >> >>> Yes, it is supposed to do the service, given a >> >>> CAS it produces a child CAS and and this CAS continue in the flow, >> >>> that are other primitive annotators. And I expected that the child >> >>> CAS >> >>> will sent to UIMA-AS client, but not do. >> >>> >> >>> the scenario: >> >>> 1- Start Broker and UIMA-AS aggregate service >> >>> 2- Client sends a CAS to UIMA-AS aggregate service to verify >> >>> everything is fine. The process is correct. >> >>> >> >>> 3- Restart Broker (without restart the service). UIMA-AS log show : >> >>> "Jms Listener Failed." .... >> >>> 4- Client sends a CAS to UIMA-AS aggregate service >> >>> >> >>> UIMA-AS client receive the parent CAS, but not child CAS. >> >>> >> >>> Thank for you quick reply >> >>> >> >>> 2016-09-22 9:48 GMT-04:00, Jaroslaw Cwiklik : >> >>> > Can you clarify if the UIMA-AS service is a Cas Multiplier. Meaning >> >>> given a >> >>> > CAS it produces child CASes >> >>> > which are then sent to UIMA-AS client. Is this what this service is >> >>> > supposed to do if the broker is running? >> >>> > >> >>> > Can you confirm the scenario: >> >>> > 1) Client sends a CAS to UIMA-AS aggregate service >> >>> > 2) Service receives the CAS >> >>> > 3) Broker dies >> >>> > 4) Broker is restarted >> >>> > >> >>> > Does the UIMA-AS client receive the CAS back in a reply when a >> >>> > broker >> >>> > is >> >>> > restarted? >> >>> > >> >>> > -Jerry >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > On Thu, Sep 22, 2016 at 9:33 AM, nelson rivera < >> >>> nelsonrivera12@gmail.com> >> >>> > wrote: >> >>> > >> >>> >> I will try the variants that you say. I was deepening the problem: >> >>> >> >> >>> >> I have a Aggregate AE that contains CasMultiplier, the original >> >>> >> CAS >> >>> >> is finish when it arrives to this CasMultiplier, this generates >> >>> >> new >> >>> >> CAS and is this CAS that continue in the flow of Aggregate. >> >>> >> >> >>> >> After restart broker (without restart the service of annotator) >> >>> >> and >> >>> >> send a CAS to process, the listener associated to the client api >> >>> >> uima-as (BaseUIMAAsynchronousEngine_impl) is notified when the >> >>> >> original CAS finalizes, but the new Cas generated in the >> >>> >> CasMultiplier, It is not notifies when finalizes. >> >>> >> >> >>> >> with Aggregate AE simple the log is the same, but how the original >> >>> >> CAS >> >>> >> is notifies into the listener when finalizes, not have problem >> >>> >> because is not expected for another new CAS. >> >>> >> >> >>> >> 2016-09-21 17:23 GMT-04:00, Jaroslaw Cwiklik : >> >>> >> > If possible can you download AMQ 5.14.0 (latest) and use that >> >>> >> > broker >> >>> >> > instead of the one the uima-as 2.8.1 ships with? There was a >> >>> >> > nasty >> >>> bug >> >>> >> > in >> >>> >> > AMQ related to keep-alive protocol not working. I am not sure if >> >>> >> > this >> >>> >> > is >> >>> >> > the cause but worth trying. >> >>> >> > >> >>> >> > You can download AMQ from here: >> >>> >> > http://activemq.apache.org/download.html >> >>> >> > >> >>> >> > -Jerry >> >>> >> > >> >>> >> > On Wed, Sep 21, 2016 at 4:52 PM, Jaroslaw Cwiklik >> >>> >> > >> >>> >> wrote: >> >>> >> > >> >>> >> >> Not sure what is causing this. I need to stage this with >> >>> >> >> UIMA-AS >> >>> >> >> 2.8.1. >> >>> >> I >> >>> >> >> am currently testing with trunk (latest) code. The most current >> >>> >> >> code >> >>> >> does >> >>> >> >> have newer broker. >> >>> >> >> >> >>> >> >> I've started a broker and a sample service: >> >>> >> >> >> >>> >> >> $UIMA_HOME/bin/deployAsyncService.sh >> >>> >> >> $UIMA_HOME/examples/deploy/as/ >> >>> >> >> Deploy_MeetingDetectorTAE.xml >> >>> >> >> >> >>> >> >> After the service started, I stopped the broker, waited a bit, >> and >> >>> >> >> restarted the broker. >> >>> >> >> >> >>> >> >> With UIMA-AS from trunk I see the following in the log: >> >>> >> >> >> >>> >> >> 04:21:35.496 - 1: org.apache.uima.adapter.jms.activemq. >> >>> >> >> SpringContainerDeployer.doStartListeners: INFO: Controller: >> >>> >> >> Meeting >> >>> >> >> Detector TAE Trying to Start Listener on Endpoint: >> >>> >> >> queue://MeetingDetectorTaeQueue >> >>> >> >> Selector: Command=2000 OR Command=2002 Broker: >> >>> >> >> tcp://localhost:61616 >> >>> >> >> 04:21:35.873 - 1: org.apache.uima.adapter.jms.activemq. >> >>> >> >> SpringContainerDeployer.doStartListeners: INFO: Controller: >> >>> >> >> Meeting >> >>> >> >> Detector TAE Trying to Start Listener on Endpoint: >> >>> >> >> queue://MeetingDetectorTaeQueue >> >>> >> >> Selector: Command=2001 Broker: tcp://localhost:61616 >> >>> >> >> 04:21:45.621 - 22: org.apache.uima.adapter.jms.activemq. >> >>> >> >> UimaDefaultMessageListenerContainer.onException: WARNING: >> Service: >> >>> >> >> Meeting Detector TAE Runtime Exception >> >>> >> >> 04:21:45.622 - 22: org.apache.uima.adapter.jms.activemq. >> >>> >> >> UimaDefaultMessageListenerContainer.onException: WARNING: Jms >> >>> Listener >> >>> >> >> Failed. Endpoint: MeetingDetectorTaeQueue Managed By: >> >>> >> >> tcp://localhost:61616 >> >>> >> >> Reason: javax.jms.JMSException: java.io.EOFException >> >>> >> >> 04:21:45.622 - 22: org.apache.uima.adapter.jms.activemq. >> >>> >> >> UimaDefaultMessageListenerContainer.handleListenerSetupFailure: >> >>> >> WARNING: >> >>> >> >> Uima AS Service:Meeting Detector TAE Listener Unable To Connect >> To >> >>> >> >> Broker: >> >>> >> >> tcp://localhost:61616 Retrying Until Successful ... >> >>> >> >> 04:21:45.638 - 27: org.apache.uima.adapter.jms.activemq. >> >>> >> >> UimaDefaultMessageListenerContainer.onException: WARNING: >> Service: >> >>> >> >> Meeting Detector TAE Runtime Exception >> >>> >> >> 04:21:45.639 - 27: org.apache.uima.adapter.jms.activemq. >> >>> >> >> UimaDefaultMessageListenerContainer.onException: WARNING: Jms >> >>> Listener >> >>> >> >> Failed. Endpoint: MeetingDetectorTaeQueue Managed By: >> >>> >> >> tcp://localhost:61616 >> >>> >> >> Reason: javax.jms.JMSException: java.io.EOFException >> >>> >> >> 04:21:45.639 - 27: org.apache.uima.adapter.jms.activemq. >> >>> >> >> UimaDefaultMessageListenerContainer.handleListenerSetupFailure: >> >>> >> WARNING: >> >>> >> >> Uima AS Service:Meeting Detector TAE Listener Unable To Connect >> To >> >>> >> >> Broker: >> >>> >> >> tcp://localhost:61616 Retrying Until Successful ... >> >>> >> >> 04:22:07.439 - 22: org.apache.uima.adapter.jms.activemq. >> >>> >> >> UimaDefaultMessageListenerContainer.handleListenerSetupFailure: >> >>> >> WARNING: >> >>> >> >> Uima AS Service:Meeting Detector TAE Listener Recovered >> Connection >> >>> to >> >>> >> >> Broker: tcp://localhost:61616 - Ready to Process Again >> >>> >> >> 04:22:07.454 - 27: org.apache.uima.adapter.jms.activemq. >> >>> >> >> UimaDefaultMessageListenerContainer.handleListenerSetupFailure: >> >>> >> WARNING: >> >>> >> >> Uima AS Service:Meeting Detector TAE Listener Recovered >> Connection >> >>> to >> >>> >> >> Broker: tcp://localhost:61616 - Ready to Process Again >> >>> >> >> >> >>> >> >> >> >>> >> >> I will repeat the above test with v 2.8.1 >> >>> >> >> >> >>> >> >> -Jerry >> >>> >> >> >> >>> >> >> >> >>> >> >> On Wed, Sep 21, 2016 at 4:34 PM, nelson rivera < >> >>> >> nelsonrivera12@gmail.com> >> >>> >> >> wrote: >> >>> >> >> >> >>> >> >>> I'm using UIMA-AS 2.8.1, any idea that what happen? >> >>> >> >>> >> >>> >> >>> 2016-09-21 16:09 GMT-04:00, Jaroslaw Cwiklik > >: >> >>> >> >>> > Which version of UIMA-AS are you using? >> >>> >> >>> > >> >>> >> >>> > -Jerry >> >>> >> >>> > >> >>> >> >>> > On Wed, Sep 21, 2016 at 3:42 PM, nelson rivera < >> >>> >> >>> nelsonrivera12@gmail.com> >> >>> >> >>> > wrote: >> >>> >> >>> > >> >>> >> >>> >> When start ActiveMQ and deploys an annotator as service, >> >>> >> >>> >> processing >> >>> >> >>> >> is >> >>> >> >>> >> executed correctly, but when the broker is stopped and >> >>> >> >>> >> subsequently >> >>> >> >>> >> restarted. UIMA-AS log show the lines below and after this >> >>> >> >>> >> the >> >>> >> >>> >> api >> >>> >> >>> >> client uima-as is not notified in the listener when a cas >> >>> >> >>> >> is >> >>> >> >>> >> processed, any more. >> >>> >> >>> >> can help me?: >> >>> >> >>> >> >> >>> >> >>> >> 02:08:25.570 - 14: >> >>> >> >>> >> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListe >> >>> nerCont >> >>> >> >>> >> ainer.onException: >> >>> >> >>> >> ADVERTENCIA: Service: Aggregate Cluster Analyzer Runtime >> >>> Exception >> >>> >> >>> >> 02:08:25.570 - 14: >> >>> >> >>> >> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListe >> >>> nerCont >> >>> >> >>> >> ainer.onException: >> >>> >> >>> >> ADVERTENCIA: Jms Listener Failed. Endpoint: >> >>> >> XClusterAnalyzerAggregate >> >>> >> >>> >> Managed By: tcp://localhost:61616 Reason: >> >>> javax.jms.JMSException: >> >>> >> >>> >> java.io.EOFException >> >>> >> >>> >> 02:08:25.570 - 14: >> >>> >> >>> >> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListe >> >>> >> >>> nerContainer. >> >>> >> >>> >> handleListenerSetupFailure: >> >>> >> >>> >> ADVERTENCIA: Uima AS Service:Aggregate Cluster Analyzer >> >>> >> >>> >> Listener >> >>> >> >>> >> Unable To Connect To Broker: tcp://localhost:61616 Retrying >> >>> >> >>> >> ... >> >>> >> >>> >> QueueFailure: ADVERTENCIA: Jms Listener Failed. Endpoint: >> >>> >> >>> >> temp-queue://ID:localhost-H81-M1-33610-1474481197190-1:1:1 >> >>> Managed >> >>> >> >>> >> By: >> >>> >> >>> >> tcp://localhost:61616 Reason: javax.jms.JMSException: >> >>> >> >>> >> java.io.EOFException >> >>> >> >>> >> 02:09:02.222 - 14: >> >>> >> >>> >> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListe >> >>> >> >>> nerContainer. >> >>> >> >>> >> handleListenerSetupFailure: >> >>> >> >>> >> ADVERTENCIA: Uima AS Service:Aggregate Cluster Analyzer >> >>> >> >>> >> Listener >> >>> >> >>> >> Established Connection to Broker: tcp://localhost:61616 >> >>> >> >>> >> >> >>> >> >>> >> 02:09:02.317 - 14: >> >>> >> >>> >> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListe >> >>> nerCont >> >>> >> >>> >> ainer.onException: >> >>> >> >>> >> ADVERTENCIA: Service: Aggregate Cluster Analyzer Runtime >> >>> Exception >> >>> >> >>> >> 02:09:02.317 - 14: >> >>> >> >>> >> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListe >> >>> nerCont >> >>> >> >>> >> ainer.onException: >> >>> >> >>> >> ADVERTENCIA: Jms Listener Failed. Endpoint: >> >>> >> XClusterAnalyzerAggregate >> >>> >> >>> >> Managed By: tcp://localhost:61616 Reason: >> >>> >> >>> >> org.apache.activemq.ConnectionClosedException: The >> connection >> >>> is >> >>> >> >>> >> already closed02:09:02.356 - 15: >> >>> >> >>> >> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListe >> >>> nerCont >> >>> >> >>> >> ainer.onException: >> >>> >> >>> >> ADVERTENCIA: Service: Aggregate Cluster Analyzer Runtime >> >>> Exception >> >>> >> >>> >> 02:09:02.357 - 15: >> >>> >> >>> >> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListe >> >>> nerCont >> >>> >> >>> >> ainer.onException: >> >>> >> >>> >> ADVERTENCIA: Jms Listener Failed. Endpoint: >> >>> >> XClusterAnalyzerAggregate >> >>> >> >>> >> Managed By: tcp://localhost:61616 Reason: >> >>> >> >>> >> javax.jms.IllegalStateException: The Consumer is closed. >> >>> >> >>> >> >> >>> >> >>> > >> >>> >> >>> >> >>> >> >> >> >>> >> >> >> >>> >> > >> >>> >> >> >>> > >> >>> >> >> >> >> >> > >> >