Return-Path: X-Original-To: apmail-uima-user-archive@www.apache.org Delivered-To: apmail-uima-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3F74F173EE for ; Fri, 20 Feb 2015 14:53:42 +0000 (UTC) Received: (qmail 17350 invoked by uid 500); 20 Feb 2015 14:53:41 -0000 Delivered-To: apmail-uima-user-archive@uima.apache.org Received: (qmail 17305 invoked by uid 500); 20 Feb 2015 14:53:41 -0000 Mailing-List: contact user-help@uima.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@uima.apache.org Delivered-To: mailing list user@uima.apache.org Received: (qmail 17289 invoked by uid 99); 20 Feb 2015 14:53:41 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Feb 2015 14:53:41 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of uimaee@gmail.com designates 74.125.82.41 as permitted sender) Received: from [74.125.82.41] (HELO mail-wg0-f41.google.com) (74.125.82.41) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Feb 2015 14:53:37 +0000 Received: by mail-wg0-f41.google.com with SMTP id b13so13622588wgh.0 for ; Fri, 20 Feb 2015 06:51:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=CxBlgVS/feYU9FSAkeKT3HOMMT1+6ZUa+agdODuRcTk=; b=pv3dEIF5ipB+bfvmJnRH/aqef/Q7tJ86BWXja4YPxpMFN+zkGsFfT2+F0Cg8SDp7Cg UXpslzS4+2Lfxl2tN2pJ4rBKdLadBl6BDvDdKkag96hvQCFRIDkN9syG383erY0m8e2V XpzWMsogXoWMMlwX0cc/KFWSIKn/GHOAvHbEYiBV7wMMmyZCzXTD9QPt6T9kTvdtfyl+ aJwIsq5EacouOAxKpZ25NjZFLOzW5iaOCl35W5dBeQfsPdXKfi6FGSLaHsOZWcU1WbyJ cUq/UGMxZjRH7inWX9e0Az3P1mpRrZ71myGUJge3Evvi4bp56gwUEzJa8E9Gwy2JBQEE JtZw== MIME-Version: 1.0 X-Received: by 10.180.14.196 with SMTP id r4mr24778185wic.77.1424443860026; Fri, 20 Feb 2015 06:51:00 -0800 (PST) Received: by 10.27.15.6 with HTTP; Fri, 20 Feb 2015 06:50:59 -0800 (PST) In-Reply-To: <54E6B775.4050301@orkash.com> References: <532BD841.2010705@orkash.com> <5332A701.6070301@orkash.com> <5332DAB3.6030808@orkash.com> <5333B886.9070709@orkash.com> <53350729.2030807@orkash.com> <533946FE.2000204@orkash.com> <54E4393C.2010708@orkash.com> <54E6B775.4050301@orkash.com> Date: Fri, 20 Feb 2015 09:50:59 -0500 Message-ID: Subject: Re: Ducc Problems From: Jaroslaw Cwiklik To: user@uima.apache.org Content-Type: multipart/alternative; boundary=f46d04138c7d478bed050f862ef7 X-Virus-Checked: Checked by ClamAV on apache.org --f46d04138c7d478bed050f862ef7 Content-Type: text/plain; charset=UTF-8 Reshu, can you confirm if you are running UIMA-AS 2.6.0 or earlier. You can look at the log for something similar to this: +------------------------------------------------------------------ + Service Name:Person Title Annotator + Service Queue Name:PersonTitleAnnotatorQueue + Service Start Time:20 Feb 2015 09:25:43 *+ UIMA AS Version:2.6.0 <-----------------------------------------------------------------------* + UIMA Core Version:2.6.0 + OS Name:Linux + OS Version:2.6.32-279.el6.x86_64 + OS Architecture:amd64 + OS CPU Count:4 + JVM Vendor:IBM Corporation + JVM Name:IBM J9 VM + JVM Version:2.6 Jerry On Thu, Feb 19, 2015 at 11:26 PM, reshu.agarwal wrote: > > Dear Cwiklik, > > There is only 2 seconds delay between the last log message and > org.apache.uima.aae.controller.PrimitiveAnalysisEngineController_impl > quiesceAndStop. > > Please have a look on the logs: > > Process Received a Message. Is Process target for message:true. Target >>>>>>>>>> PID:22640 >>>>>>>>>> >>>>>>>>> configFactory.stop() - stopped route:mina:tcp://localhost: >>>> 52449?transferExchange=true&sync=false >>>> >>> Feb 19, 2015 5:39:54 PM org.apache.uima.aae.controller. > PrimitiveAnalysisEngineController_impl quiesceAndStop > INFO: Stopping Controller: ducc.jd.queue.13202 > Quiescing UIMA-AS Service. Remaining Number of CASes to Process:0 > Feb 19, 2015 5:39:54 PM org.apache.uima.adapter.jms.activemq.JmsInputChannel > stopChannel > INFO: Stopping Service JMS Transport. Service: ducc.jd.queue.13202 > ShutdownNow false > Feb 19, 2015 5:39:54 PM org.apache.uima.adapter.jms.activemq.JmsInputChannel > stopChannel > INFO: Controller: ducc.jd.queue.13202 Stopped Listener on Endpoint: > queue://ducc.jd.queue.13202 Selector: Selector:Command=2000 OR > Command=2002. > Feb 19, 2015 5:39:54 PM org.apache.uima.adapter.jms.activemq.JmsInputChannel > stopChannel > INFO: Stopping Service JMS Transport. Service: ducc.jd.queue.13202 > ShutdownNow false > Feb 19, 2015 5:39:54 PM org.apache.uima.adapter.jms.activemq.JmsInputChannel > stopChannel > INFO: Controller: ducc.jd.queue.13202 Stopped Listener on Endpoint: > queue://ducc.jd.queue.13202 Selector: Selector:Command=2001. > Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.activemq.JmsInputChannel > stopChannel > INFO: Stopping Service JMS Transport. Service: ducc.jd.queue.13202 > ShutdownNow true > Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.activemq.JmsInputChannel > stopChannel > INFO: Controller: ducc.jd.queue.13202 Stopped Listener on Endpoint: > queue://ducc.jd.queue.13202 Selector: Selector:Command=2000 OR > Command=2002. > Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.activemq.JmsInputChannel > stopChannel > INFO: Stopping Service JMS Transport. Service: ducc.jd.queue.13202 > ShutdownNow true > Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.activemq.JmsInputChannel > stopChannel > INFO: Controller: ducc.jd.queue.13202 Stopped Listener on Endpoint: > queue://ducc.jd.queue.13202 Selector: Selector:Command=2001. > Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.activemq.JmsInputChannel > stopChannel > INFO: Stopping Service JMS Transport. Service: ducc.jd.queue.13202 > ShutdownNow false > Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.activemq.JmsInputChannel > stopChannel > INFO: Controller: ducc.jd.queue.13202 Stopped Listener on Endpoint: > queue://ducc.jd.queue.13202 Selector: Selector:Command=2000 OR > Command=2002. > Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.activemq.JmsInputChannel > stopChannel > INFO: Stopping Service JMS Transport. Service: ducc.jd.queue.13202 > ShutdownNow false > Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.activemq.JmsInputChannel > stopChannel > INFO: Controller: ducc.jd.queue.13202 Stopped Listener on Endpoint: > queue://ducc.jd.queue.13202 Selector: Selector:Command=2001. > Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.activemq.JmsInputChannel > stopChannel > INFO: Stopping Service JMS Transport. Service: ducc.jd.queue.13202 > ShutdownNow true > Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.activemq.JmsInputChannel > stopChannel > INFO: Controller: ducc.jd.queue.13202 Stopped Listener on Endpoint: > queue://ducc.jd.queue.13202 Selector: Selector:Command=2000 OR > Command=2002. > Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.activemq.JmsInputChannel > stopChannel > INFO: Stopping Service JMS Transport. Service: ducc.jd.queue.13202 > ShutdownNow true > Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.activemq.JmsInputChannel > stopChannel > INFO: Controller: ducc.jd.queue.13202 Stopped Listener on Endpoint: > queue://ducc.jd.queue.13202 Selector: Selector:Command=2001. > UIMA-AS Service is Stopping, All CASes Have Been Processed > Feb 19, 2015 5:39:56 PM org.apache.uima.aae.controller. > PrimitiveAnalysisEngineController_impl stop > INFO: Stopping Controller: ducc.jd.queue.13202 > Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.activemq.JmsInputChannel > stopChannel > INFO: Stopping Service JMS Transport. Service: ducc.jd.queue.13202 > ShutdownNow true > Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.activemq.JmsInputChannel > stopChannel > INFO: Controller: ducc.jd.queue.13202 Stopped Listener on Endpoint: > queue://ducc.jd.queue.13202 Selector: Selector:Command=2000 OR > Command=2002. > Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.activemq.JmsInputChannel > stopChannel > INFO: Stopping Service JMS Transport. Service: ducc.jd.queue.13202 > ShutdownNow true > Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.activemq.JmsInputChannel > stopChannel > INFO: Controller: ducc.jd.queue.13202 Stopped Listener on Endpoint: > queue://ducc.jd.queue.13202 Selector: Selector:Command=2001. > Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.activemq.JmsOutputChannel > stop > INFO: Controller: ducc.jd.queue.13202 Output Channel Shutdown Completed > > Thanks Reshu. > > > > On 02/20/2015 12:40 AM, Jaroslaw Cwiklik wrote: > >> One possible explanation for destroy() not getting called is that a >> process >> (JP) may be still working on a CAS when Ducc deallocates the process. Ducc >> first asks the process to quiesce and stop and allows it 1 minute to >> terminate on its own. If this does not happen, Ducc kills the process via >> kill -9. In such case the process will be clobbered and destroy() methods >> in UIMA-AS are not called. >> There should be some evidence in JP logs at the very end. Look for >> something like this: >> >> Process Received a Message. Is Process target for message:true. >>>>>>>>>>> >>>>>>>>>> Target PID:27520 >> >>> configFactory.stop() - stopped >>>>> >>>> route:mina:tcp://localhost:49338?transferExchange=true&sync=false >> 01:56:22.735 - 94: >> org.apache.uima.aae.controller.PrimitiveAnalysisEngineControl >> ler_impl.quiesceAndStop: >> INFO: Stopping Controller: ducc.jd.queue.226091 >> Quiescing UIMA-AS Service. Remaining Number of CASes to Process:0 >> >> Look at the timestamp of >>>>>>>>> Process Received a Message. Is Process >> target for message:true. >> and compare it to a timestamp of the last log message. Does it look like >> there is a long delay? >> >> >> Jerry >> >> On Wed, Feb 18, 2015 at 2:03 AM, reshu.agarwal >> wrote: >> >> Dear Eddie, >>> >>> This problem has been resolved by using destroy method in ducc version >>> 1.0.0 but when I upgrade my ducc version from 1.0.0 to 1.1.0 DUCC didn't >>> call the destroy method. >>> >>> It also do not call the stop method of CollectionReader as well as >>> finalize method of any java class as well as destroy/ >>> collectionProcessComplete >>> method of cas consumer. >>> >>> I want to close my connection to Database after completion of job as well >>> as want to use batch processing at cas consumer level like >>> PersonTitleDBWriterCasConsumer. >>> >>> Thanks in advanced. >>> >>> Reshu. >>> >>> >>> >>> >>> On 03/31/2014 04:14 PM, reshu.agarwal wrote: >>> >>> On 03/28/2014 05:28 PM, Eddie Epstein wrote: >>>> >>>> Another alternative would be to do the final flush in the Cas >>>>> consumer's >>>>> destroy method. >>>>> >>>>> Another issue to be aware of, in order to balance resources between >>>>> jobs, >>>>> DUCC uses preemption of job processes scheduled in a "fair-share" >>>>> class. >>>>> This may not be acceptable for jobs which are doing incremental >>>>> commits. >>>>> The solution is to schedule the job in a non-preemptable class. >>>>> >>>>> >>>>> On Fri, Mar 28, 2014 at 1:22 AM, reshu.agarwal < >>>>> reshu.agarwal@orkash.com >>>>> >>>>>> wrote: >>>>>> >>>>> On 03/28/2014 01:28 AM, Eddie Epstein wrote: >>>>> >>>>>> Hi Reshu, >>>>>> >>>>>>> The Job model in DUCC is for the Collection Reader to send "work item >>>>>>> CASes", where a work item represents a collection of work to be done >>>>>>> by a >>>>>>> Job Process. For example, a work item could be a file or a subset of >>>>>>> a >>>>>>> file >>>>>>> that contains many documents, where each document would be >>>>>>> individually >>>>>>> put >>>>>>> into a CAS by the Cas Multiplier in the Job Process. >>>>>>> >>>>>>> DUCC is designed so that after processing the "mini-collection" >>>>>>> represented >>>>>>> by the work item, the Cas Consumer should flush any data. This is >>>>>>> done by >>>>>>> routing the "work item CAS" to the Cas Consumer, after all work item >>>>>>> documents are completed, at which point the CC does the flush. >>>>>>> >>>>>>> The sample code described in >>>>>>> http://uima.apache.org/d/uima-ducc-1.0.0/duccbook.html#x1-1380009 >>>>>>> uses >>>>>>> the >>>>>>> work item CAS to flush data in exactly this way. >>>>>>> >>>>>>> Note that the PersonTitleDBWriterCasConsumer is doing a flush (a >>>>>>> commit) >>>>>>> in >>>>>>> the process method after every 50 documents. >>>>>>> >>>>>>> Regards >>>>>>> Eddie >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Mar 27, 2014 at 1:35 AM, reshu.agarwal < >>>>>>> reshu.agarwal@orkash.com> >>>>>>> wrote: >>>>>>> >>>>>>> On 03/26/2014 11:34 PM, Eddie Epstein wrote: >>>>>>> >>>>>>> Hi Reshu, >>>>>>>> >>>>>>>> The collectionProcessingComplete() method in UIMA-AS has a >>>>>>>>> limitation: a >>>>>>>>> Collection Processing Complete request sent to the UIMA-AS Analysis >>>>>>>>> Service >>>>>>>>> is cascaded down to all delegates; however, if a particular >>>>>>>>> delegate >>>>>>>>> is >>>>>>>>> scaled-out, only one of the instances of the delegate will get this >>>>>>>>> call. >>>>>>>>> >>>>>>>>> Since DUCC is using UIMA-AS to scale out the Job processes, it has >>>>>>>>> no >>>>>>>>> way >>>>>>>>> to deliver a CPC to all instances. >>>>>>>>> >>>>>>>>> The applications we have been running on DUCC have used the Work >>>>>>>>> Item >>>>>>>>> CAS >>>>>>>>> as a signal to CAS consumers to do CPC level processing. That is >>>>>>>>> discussed >>>>>>>>> in the first reference above, in the paragraph "Flushing Cached >>>>>>>>> Data". >>>>>>>>> >>>>>>>>> Eddie >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Mar 26, 2014 at 9:48 AM, reshu.agarwal < >>>>>>>>> reshu.agarwal@orkash.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> On 03/26/2014 06:43 PM, Eddie Epstein wrote: >>>>>>>>> >>>>>>>>> Are you using standard UIMA interface code to Solr? If so, >>>>>>>>> which >>>>>>>>> >>>>>>>>>> Cas >>>>>>>>>> >>>>>>>>>> Consumer? >>>>>>>>>> >>>>>>>>>>> Taking at quick look at the source code for SolrCASConsumer, the >>>>>>>>>>> batch >>>>>>>>>>> and >>>>>>>>>>> collection process complete methods appear to do nothing. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Eddie >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Mar 26, 2014 at 6:08 AM, reshu.agarwal < >>>>>>>>>>> reshu.agarwal@orkash.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> On 03/21/2014 11:42 AM, reshu.agarwal wrote: >>>>>>>>>>> >>>>>>>>>>> Hence we can not attempt batch processing in cas consumer >>>>>>>>>>> and >>>>>>>>>>> it >>>>>>>>>>> >>>>>>>>>>> increases our process timing. Is there any other option for >>>>>>>>>>>> that or >>>>>>>>>>>> >>>>>>>>>>>> is >>>>>>>>>>>>> it a >>>>>>>>>>>>> bug in DUCC? >>>>>>>>>>>>> >>>>>>>>>>>>> Please reply on this problem as if I am sending document >>>>>>>>>>>>> in >>>>>>>>>>>>> solr >>>>>>>>>>>>> one by >>>>>>>>>>>>> >>>>>>>>>>>>> one by cas consumer without using batch process and >>>>>>>>>>>>> committing >>>>>>>>>>>>> >>>>>>>>>>>>> solr. It >>>>>>>>>>>> is >>>>>>>>>>>> not optimum way to use this. Why ducc is not calling collection >>>>>>>>>>>> Process >>>>>>>>>>>> Complete method of Cas Consumer? And If I want to do that then >>>>>>>>>>>> What >>>>>>>>>>>> is >>>>>>>>>>>> the >>>>>>>>>>>> way to do this? >>>>>>>>>>>> >>>>>>>>>>>> I am not able to find any thing about this in DUCC book. >>>>>>>>>>>> >>>>>>>>>>>> Thanks in Advanced. >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Reshu Agarwal >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hi Eddie, >>>>>>>>>>>> >>>>>>>>>>>> I am not using standard UIMA interface code to Solr. I >>>>>>>>>>>> create my >>>>>>>>>>>> >>>>>>>>>>>> own Cas >>>>>>>>>>> >>>>>>>>>>> Consumer. I will take a look on that too. But the problem is >>>>>>>>>>> not >>>>>>>>>>> >>>>>>>>>> for >>>>>>>>>> particularly to use solr, I can use any source to store my >>>>>>>>>> output. I >>>>>>>>>> want >>>>>>>>>> to do batch processing and want to use collectionProcessComplete. >>>>>>>>>> Why >>>>>>>>>> DUCC >>>>>>>>>> is not calling it? I check it with UIMA AS also and my cas >>>>>>>>>> consumer >>>>>>>>>> is >>>>>>>>>> working fine with it and also performing batch processing. >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Thanks, >>>>>>>>>> Reshu Agarwal >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hi Eddie, >>>>>>>>>> >>>>>>>>>> I am using cas consumer similar to apache uima example: >>>>>>>>>> >>>>>>>>> "apache-uima/examples/src/org/apache/uima/examples/cpe/ >>>>>>>> PersonTitleDBWriterCasConsumer.java" >>>>>>>> >>>>>>>> -- >>>>>>>> Thanks, >>>>>>>> Reshu Agarwal >>>>>>>> >>>>>>>> >>>>>>>> Hi Eddie, >>>>>>>> >>>>>>>> You are right I know this fact. PersonTitleDBWriterCasConsumer is >>>>>>> >>>>>> doing a >>>>>> flush (a commit) in the process method after every 50 documents and if >>>>>> less >>>>>> then 50 documents in cas it will do commit or flush by >>>>>> collectionProcessComplete method. So, If it is not called then those >>>>>> documents can not be committed. That is why I want ducc calls this >>>>>> method. >>>>>> >>>>>> -- >>>>>> Thanks, >>>>>> Reshu Agarwal >>>>>> >>>>>> >>>>>> Hi, >>>>>> >>>>> Destroy method worked for me. It did the same what I wanted from >>>> CollectionProcessComplete method. >>>> >>>> >>>> > --f46d04138c7d478bed050f862ef7--