uima-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eddie Epstein <eaepst...@gmail.com>
Subject Re: Ducc Problems
Date Wed, 04 Mar 2015 01:36:46 GMT
Reshu,

With preemption active there is no way to guarantee that the destroy method
will be called in a Job Process. A job submitted to a non-preemptable
class, like "fixed", will not have any preemption.

Eddie


On Sun, Feb 22, 2015 at 11:11 PM, reshu.agarwal <reshu.agarwal@orkash.com>
wrote:

> I am running Uima-AS 2.6.0. Please have a look on logs:
>
> Feb 23, 2015 9:37:44 AM org.apache.uima.adapter.jms.service.UIMA_Service
> initialize(67)
> INFO: UIMA-AS version 2.6.0
>
>
>
> On 02/20/2015 08:20 PM, Jaroslaw Cwiklik wrote:
>
>> Reshu, can you confirm if you are running UIMA-AS 2.6.0 or earlier. You
>> can
>> look at the log for something similar to this:
>>
>> +------------------------------------------------------------------
>> + Service Name:Person Title Annotator
>> + Service Queue Name:PersonTitleAnnotatorQueue
>> + Service Start Time:20 Feb 2015 09:25:43
>> *+ UIMA AS Version:2.6.0
>>   <-----------------------------------------------------------
>> ------------*
>> + UIMA Core Version:2.6.0
>> + OS Name:Linux
>> + OS Version:2.6.32-279.el6.x86_64
>> + OS Architecture:amd64
>> + OS CPU Count:4
>> + JVM Vendor:IBM Corporation
>> + JVM Name:IBM J9 VM
>> + JVM Version:2.6
>>
>> Jerry
>>
>> On Thu, Feb 19, 2015 at 11:26 PM, reshu.agarwal <reshu.agarwal@orkash.com
>> >
>> wrote:
>>
>>  Dear Cwiklik,
>>>
>>> There is only 2 seconds delay between the last log message and
>>> org.apache.uima.aae.controller.PrimitiveAnalysisEngineController_impl
>>> quiesceAndStop.
>>>
>>> Please have a look on the logs:
>>>
>>>   Process Received a Message. Is Process target for message:true. Target
>>>
>>>> PID:22640
>>>>>>>>>>>>
>>>>>>>>>>>>  configFactory.stop() - stopped route:mina:tcp://localhost:
>>>>>>>>>>>
>>>>>>>>>> 52449?transferExchange=true&sync=false
>>>>>>
>>>>>>  Feb 19, 2015 5:39:54 PM org.apache.uima.aae.controller.
>>>>>
>>>> PrimitiveAnalysisEngineController_impl quiesceAndStop
>>> INFO: Stopping Controller: ducc.jd.queue.13202
>>> Quiescing UIMA-AS Service. Remaining Number of CASes to Process:0
>>> Feb 19, 2015 5:39:54 PM org.apache.uima.adapter.jms.
>>> activemq.JmsInputChannel
>>> stopChannel
>>> INFO: Stopping Service JMS Transport. Service: ducc.jd.queue.13202
>>> ShutdownNow false
>>> Feb 19, 2015 5:39:54 PM org.apache.uima.adapter.jms.
>>> activemq.JmsInputChannel
>>> stopChannel
>>> INFO: Controller: ducc.jd.queue.13202 Stopped Listener on Endpoint:
>>> queue://ducc.jd.queue.13202 Selector:  Selector:Command=2000 OR
>>> Command=2002.
>>> Feb 19, 2015 5:39:54 PM org.apache.uima.adapter.jms.
>>> activemq.JmsInputChannel
>>> stopChannel
>>> INFO: Stopping Service JMS Transport. Service: ducc.jd.queue.13202
>>> ShutdownNow false
>>> Feb 19, 2015 5:39:54 PM org.apache.uima.adapter.jms.
>>> activemq.JmsInputChannel
>>> stopChannel
>>> INFO: Controller: ducc.jd.queue.13202 Stopped Listener on Endpoint:
>>> queue://ducc.jd.queue.13202 Selector:  Selector:Command=2001.
>>> Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.
>>> activemq.JmsInputChannel
>>> stopChannel
>>> INFO: Stopping Service JMS Transport. Service: ducc.jd.queue.13202
>>> ShutdownNow true
>>> Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.
>>> activemq.JmsInputChannel
>>> stopChannel
>>> INFO: Controller: ducc.jd.queue.13202 Stopped Listener on Endpoint:
>>> queue://ducc.jd.queue.13202 Selector:  Selector:Command=2000 OR
>>> Command=2002.
>>> Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.
>>> activemq.JmsInputChannel
>>> stopChannel
>>> INFO: Stopping Service JMS Transport. Service: ducc.jd.queue.13202
>>> ShutdownNow true
>>> Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.
>>> activemq.JmsInputChannel
>>> stopChannel
>>> INFO: Controller: ducc.jd.queue.13202 Stopped Listener on Endpoint:
>>> queue://ducc.jd.queue.13202 Selector:  Selector:Command=2001.
>>> Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.
>>> activemq.JmsInputChannel
>>> stopChannel
>>> INFO: Stopping Service JMS Transport. Service: ducc.jd.queue.13202
>>> ShutdownNow false
>>> Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.
>>> activemq.JmsInputChannel
>>> stopChannel
>>> INFO: Controller: ducc.jd.queue.13202 Stopped Listener on Endpoint:
>>> queue://ducc.jd.queue.13202 Selector:  Selector:Command=2000 OR
>>> Command=2002.
>>> Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.
>>> activemq.JmsInputChannel
>>> stopChannel
>>> INFO: Stopping Service JMS Transport. Service: ducc.jd.queue.13202
>>> ShutdownNow false
>>> Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.
>>> activemq.JmsInputChannel
>>> stopChannel
>>> INFO: Controller: ducc.jd.queue.13202 Stopped Listener on Endpoint:
>>> queue://ducc.jd.queue.13202 Selector:  Selector:Command=2001.
>>> Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.
>>> activemq.JmsInputChannel
>>> stopChannel
>>> INFO: Stopping Service JMS Transport. Service: ducc.jd.queue.13202
>>> ShutdownNow true
>>> Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.
>>> activemq.JmsInputChannel
>>> stopChannel
>>> INFO: Controller: ducc.jd.queue.13202 Stopped Listener on Endpoint:
>>> queue://ducc.jd.queue.13202 Selector:  Selector:Command=2000 OR
>>> Command=2002.
>>> Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.
>>> activemq.JmsInputChannel
>>> stopChannel
>>> INFO: Stopping Service JMS Transport. Service: ducc.jd.queue.13202
>>> ShutdownNow true
>>> Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.
>>> activemq.JmsInputChannel
>>> stopChannel
>>> INFO: Controller: ducc.jd.queue.13202 Stopped Listener on Endpoint:
>>> queue://ducc.jd.queue.13202 Selector:  Selector:Command=2001.
>>> UIMA-AS Service is Stopping, All CASes Have Been Processed
>>> Feb 19, 2015 5:39:56 PM org.apache.uima.aae.controller.
>>> PrimitiveAnalysisEngineController_impl stop
>>> INFO: Stopping Controller: ducc.jd.queue.13202
>>> Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.
>>> activemq.JmsInputChannel
>>> stopChannel
>>> INFO: Stopping Service JMS Transport. Service: ducc.jd.queue.13202
>>> ShutdownNow true
>>> Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.
>>> activemq.JmsInputChannel
>>> stopChannel
>>> INFO: Controller: ducc.jd.queue.13202 Stopped Listener on Endpoint:
>>> queue://ducc.jd.queue.13202 Selector:  Selector:Command=2000 OR
>>> Command=2002.
>>> Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.
>>> activemq.JmsInputChannel
>>> stopChannel
>>> INFO: Stopping Service JMS Transport. Service: ducc.jd.queue.13202
>>> ShutdownNow true
>>> Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.
>>> activemq.JmsInputChannel
>>> stopChannel
>>> INFO: Controller: ducc.jd.queue.13202 Stopped Listener on Endpoint:
>>> queue://ducc.jd.queue.13202 Selector:  Selector:Command=2001.
>>> Feb 19, 2015 5:39:56 PM org.apache.uima.adapter.jms.
>>> activemq.JmsOutputChannel
>>> stop
>>> INFO: Controller: ducc.jd.queue.13202 Output Channel Shutdown Completed
>>>
>>> Thanks Reshu.
>>>
>>>
>>>
>>> On 02/20/2015 12:40 AM, Jaroslaw Cwiklik wrote:
>>>
>>>  One possible explanation for destroy() not getting called is that a
>>>> process
>>>> (JP) may be still working on a CAS when Ducc deallocates the process.
>>>> Ducc
>>>> first asks the process to quiesce and stop and allows it 1 minute to
>>>> terminate on its own. If this does not happen, Ducc kills the process
>>>> via
>>>> kill -9. In such case the process will be clobbered and destroy()
>>>> methods
>>>> in UIMA-AS are not called.
>>>> There should be some evidence in JP logs at the very end. Look for
>>>> something like this:
>>>>
>>>>   Process Received a Message. Is Process target for message:true.
>>>>
>>>>> Target PID:27520
>>>>>>>>>>>>
>>>>>>>>>>> configFactory.stop() - stopped
>>>>>
>>>>>> route:mina:tcp://localhost:49338?transferExchange=true&sync=false
>>>>>>
>>>>> 01:56:22.735 - 94:
>>>> org.apache.uima.aae.controller.PrimitiveAnalysisEngineControl
>>>> ler_impl.quiesceAndStop:
>>>> INFO: Stopping Controller: ducc.jd.queue.226091
>>>> Quiescing UIMA-AS Service. Remaining Number of CASes to Process:0
>>>>
>>>> Look at the timestamp of >>>>>>>>> Process Received
a Message. Is
>>>> Process
>>>> target for message:true.
>>>> and compare it to a timestamp of the last log message. Does it look like
>>>> there is a long delay?
>>>>
>>>>
>>>> Jerry
>>>>
>>>> On Wed, Feb 18, 2015 at 2:03 AM, reshu.agarwal <
>>>> reshu.agarwal@orkash.com>
>>>> wrote:
>>>>
>>>>   Dear Eddie,
>>>>
>>>>> This problem has been resolved by using destroy method in ducc version
>>>>> 1.0.0 but when I upgrade my ducc version from 1.0.0 to 1.1.0 DUCC
>>>>> didn't
>>>>> call the destroy method.
>>>>>
>>>>> It also do not call the stop method of CollectionReader as well as
>>>>> finalize method of any java class as well as destroy/
>>>>> collectionProcessComplete
>>>>> method of cas consumer.
>>>>>
>>>>> I want to close my connection to Database after completion of job as
>>>>> well
>>>>> as want to use batch processing at cas consumer level like
>>>>> PersonTitleDBWriterCasConsumer.
>>>>>
>>>>> Thanks in advanced.
>>>>>
>>>>> Reshu.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 03/31/2014 04:14 PM, reshu.agarwal wrote:
>>>>>
>>>>>   On 03/28/2014 05:28 PM, Eddie Epstein wrote:
>>>>>
>>>>>>   Another alternative would be to do the final flush in the Cas
>>>>>>
>>>>>>> consumer's
>>>>>>> destroy method.
>>>>>>>
>>>>>>> Another issue to be aware of, in order to balance resources between
>>>>>>> jobs,
>>>>>>> DUCC uses preemption of job processes scheduled in a "fair-share"
>>>>>>> class.
>>>>>>> This may not be acceptable for jobs which are doing incremental
>>>>>>> commits.
>>>>>>> The solution is to schedule the job in a non-preemptable class.
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Mar 28, 2014 at 1:22 AM, reshu.agarwal <
>>>>>>> reshu.agarwal@orkash.com
>>>>>>>
>>>>>>>  wrote:
>>>>>>>>
>>>>>>>>     On 03/28/2014 01:28 AM, Eddie Epstein wrote:
>>>>>>>
>>>>>>>     Hi Reshu,
>>>>>>>>
>>>>>>>>  The Job model in DUCC is for the Collection Reader to send
"work
>>>>>>>>> item
>>>>>>>>> CASes", where a work item represents a collection of
work to be
>>>>>>>>> done
>>>>>>>>> by a
>>>>>>>>> Job Process. For example, a work item could be a file
or a subset
>>>>>>>>> of
>>>>>>>>> a
>>>>>>>>> file
>>>>>>>>> that contains many documents, where each document would
be
>>>>>>>>> individually
>>>>>>>>> put
>>>>>>>>> into a CAS by the Cas Multiplier in the Job Process.
>>>>>>>>>
>>>>>>>>> DUCC is designed so that after processing the "mini-collection"
>>>>>>>>> represented
>>>>>>>>> by the work item,  the Cas Consumer should flush any
data. This is
>>>>>>>>> done by
>>>>>>>>> routing the "work item CAS" to the Cas Consumer, after
all work
>>>>>>>>> item
>>>>>>>>> documents are completed, at which point the CC does the
flush.
>>>>>>>>>
>>>>>>>>> The sample code described in
>>>>>>>>> http://uima.apache.org/d/uima-ducc-1.0.0/duccbook.html#x1-1380009
>>>>>>>>> uses
>>>>>>>>> the
>>>>>>>>> work item CAS to flush data in exactly this way.
>>>>>>>>>
>>>>>>>>> Note that the PersonTitleDBWriterCasConsumer is doing
a flush (a
>>>>>>>>> commit)
>>>>>>>>> in
>>>>>>>>> the process method after every 50 documents.
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Eddie
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Mar 27, 2014 at 1:35 AM, reshu.agarwal <
>>>>>>>>> reshu.agarwal@orkash.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>     On 03/26/2014 11:34 PM, Eddie Epstein wrote:
>>>>>>>>>
>>>>>>>>>      Hi Reshu,
>>>>>>>>>
>>>>>>>>>>   The collectionProcessingComplete() method in UIMA-AS
has a
>>>>>>>>>>
>>>>>>>>>>> limitation: a
>>>>>>>>>>> Collection Processing Complete request sent to
the UIMA-AS
>>>>>>>>>>> Analysis
>>>>>>>>>>> Service
>>>>>>>>>>> is cascaded down to all delegates; however, if
a particular
>>>>>>>>>>> delegate
>>>>>>>>>>> is
>>>>>>>>>>> scaled-out, only one of the instances of the
delegate will get
>>>>>>>>>>> this
>>>>>>>>>>> call.
>>>>>>>>>>>
>>>>>>>>>>> Since DUCC is using UIMA-AS to scale out the
Job processes, it
>>>>>>>>>>> has
>>>>>>>>>>> no
>>>>>>>>>>> way
>>>>>>>>>>> to deliver a CPC to all instances.
>>>>>>>>>>>
>>>>>>>>>>> The applications we have been running on DUCC
have used the Work
>>>>>>>>>>> Item
>>>>>>>>>>> CAS
>>>>>>>>>>> as a signal to CAS consumers to do CPC level
processing. That is
>>>>>>>>>>> discussed
>>>>>>>>>>> in the first reference above, in the paragraph
"Flushing Cached
>>>>>>>>>>> Data".
>>>>>>>>>>>
>>>>>>>>>>> Eddie
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Mar 26, 2014 at 9:48 AM, reshu.agarwal
<
>>>>>>>>>>> reshu.agarwal@orkash.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>      On 03/26/2014 06:43 PM, Eddie Epstein wrote:
>>>>>>>>>>>
>>>>>>>>>>>       Are you using standard UIMA interface code
to Solr? If so,
>>>>>>>>>>> which
>>>>>>>>>>>
>>>>>>>>>>>  Cas
>>>>>>>>>>>>
>>>>>>>>>>>>    Consumer?
>>>>>>>>>>>>
>>>>>>>>>>>>  Taking at quick look at the source code
for SolrCASConsumer,
>>>>>>>>>>>>> the
>>>>>>>>>>>>> batch
>>>>>>>>>>>>> and
>>>>>>>>>>>>> collection process complete methods appear
to do nothing.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Eddie
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Mar 26, 2014 at 6:08 AM, reshu.agarwal
<
>>>>>>>>>>>>> reshu.agarwal@orkash.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>       On 03/21/2014 11:42 AM, reshu.agarwal
wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>        Hence we can not attempt batch
processing in cas
>>>>>>>>>>>>> consumer
>>>>>>>>>>>>> and
>>>>>>>>>>>>> it
>>>>>>>>>>>>>
>>>>>>>>>>>>>      increases our process timing. Is
there any other option
>>>>>>>>>>>>> for
>>>>>>>>>>>>>
>>>>>>>>>>>>>> that or
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>   is
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> it a
>>>>>>>>>>>>>>> bug in DUCC?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>       Please reply on this problem
as if I am sending
>>>>>>>>>>>>>>> document
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>> solr
>>>>>>>>>>>>>>> one by
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>     one by cas consumer without
using batch process and
>>>>>>>>>>>>>>> committing
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>   solr. It
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>> not optimum way to use this. Why
ducc is not calling
>>>>>>>>>>>>>> collection
>>>>>>>>>>>>>> Process
>>>>>>>>>>>>>> Complete method of Cas Consumer?
And If I want to do that then
>>>>>>>>>>>>>> What
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> way to do this?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am not able to find any thing about
this in DUCC book.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks in Advanced.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Reshu Agarwal
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>       Hi Eddie,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>     I am not using standard UIMA
interface code to Solr. I
>>>>>>>>>>>>>> create my
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>   own Cas
>>>>>>>>>>>>>>
>>>>>>>>>>>>>    Consumer. I will take a look on that
too. But the problem is
>>>>>>>>>>>>> not
>>>>>>>>>>>>>
>>>>>>>>>>>>>  for
>>>>>>>>>>>> particularly to use solr, I can use any source
to store my
>>>>>>>>>>>> output. I
>>>>>>>>>>>> want
>>>>>>>>>>>> to do batch processing and want to use
>>>>>>>>>>>> collectionProcessComplete.
>>>>>>>>>>>> Why
>>>>>>>>>>>> DUCC
>>>>>>>>>>>> is not calling it? I check it with UIMA AS
also and my cas
>>>>>>>>>>>> consumer
>>>>>>>>>>>> is
>>>>>>>>>>>> working fine with it and also performing
batch processing.
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Reshu Agarwal
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>      Hi Eddie,
>>>>>>>>>>>>
>>>>>>>>>>>>    I am using cas consumer similar to apache
uima example:
>>>>>>>>>>>>
>>>>>>>>>>>>       "apache-uima/examples/src/org/apache/uima/examples/cpe/
>>>>>>>>>>>
>>>>>>>>>> PersonTitleDBWriterCasConsumer.java"
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Thanks,
>>>>>>>>>> Reshu Agarwal
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>     Hi Eddie,
>>>>>>>>>>
>>>>>>>>>>   You are right I know this fact. PersonTitleDBWriterCasConsumer
>>>>>>>>>> is
>>>>>>>>>>
>>>>>>>>> doing a
>>>>>>>> flush (a commit) in the process method after every 50 documents
and
>>>>>>>> if
>>>>>>>> less
>>>>>>>> then 50 documents in cas it will do commit or flush by
>>>>>>>> collectionProcessComplete method. So, If it is not called
then those
>>>>>>>> documents can not be committed. That is why I want ducc calls
this
>>>>>>>> method.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Thanks,
>>>>>>>> Reshu Agarwal
>>>>>>>>
>>>>>>>>
>>>>>>>>    Hi,
>>>>>>>>
>>>>>>>>  Destroy method worked for me. It did the same what I wanted
from
>>>>>>>
>>>>>> CollectionProcessComplete method.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message