uima-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pascal Coupet" <pascal.cou...@temis.com>
Subject RE: Asynchronous UIMA (workflow) ?
Date Fri, 12 Oct 2007 13:32:52 GMT
Hi Greg,

I agree with you that human intervention may be often needed in NLP related applications.
In an editorial system by example, you may want a review and validation of categories assigned
automatically. However, I'm not sure that this should be done within an UIMA pipeline. The
UIMA framework is a middleware and is not the whole application. It looks difficult to me
to manage that if 10 docs go into a pipeline, 9 will go through at a normal pace and one will
get stuck somewhere 1 or 2 days for manual intervention. The framework is distributed and
relies on timeouts to detect errors. You will have to do something special to not fail in
error for this document and hope that the user will not forget to do the job.

If I go back to the editorial system example, it may require getting the document quickly
within the system even if some annotators did fail on it. The application can then make some
decisions depending on the missing parts (ask an editor to complete, hide the document ...)

One way to handle errors is simply to store them within the CAS. Subsequent annotators can
make decisions depending on previous errors. In your example, No entity extraction will be
make because no category is available and the annotator will log an error "unable to extract
entities..." which is different than finding no entity. The application receiving the CAS
at the end of the workflow will propose it to an editor who will select categories and then
resubmit the document to the annotation workflow to get it completed. 

I think that the whole purpose of the UIMA framework is to glue together various annotation
engines and manage properly to distribution of the work across machines. One workflow can
be seen as a meta annotator which has business meaning to your company or research center.
It can be ideally reused by different applications. So I will try to avoid as much as possible
to have application specific actions directly encoded within it. 


Pascal Coupet
Chief Technology Officer & Co-founder

1518 Walnut Street, suite 1702, Philadelphia, PA 19102, USA
 Tel:   +1 215 732 2549 ext 112 
Mob:   +1 215 609 2514
Fax:   +1 215 732 0490 

Strictly Personal and Confidential
This message may contain confidential and proprietary material for the sole use of the intended
recipient. Any review or distribution by or to others is strictly prohibited. If you are not
the intended recipient, please contact the sender and delete all copies.
-----Original Message-----
From: greg@holmberg.name [mailto:holmberg2066@comcast.net] 
Sent: Thursday, October 11, 2007 1:18 PM
To: uima-user@incubator.apache.org; uima-user@incubator.apache.org
Cc: Pascal Coupet
Subject: RE: Asynchronous UIMA (workflow) ?


I was thinking essentially the same thing: serialize the CAS to a file or database, do your
human interaction (possibly including the CAS Editor), then reload it and resume processing.

It would be nice to generalize it, rather than have two explicit analysis engines.  So a nice
enhancement to UIMA would be the ability to persist not just the CAS but the state of the
engine along with it, so that it could be stopped and restarted at any point.

For my purposes, this would be useful if say, one annotator depended on finding certain data
in the CAS from another annotator, but that earlier one failed or didn't produce the right
data, and I need a user to produce the data manually.

For example, if a taxonomy classifier runs first and a named entity extractor runs second,
and the entity extractor wants to select a name catalog to use based on the classification
("if classified biology, use biology NC, else if classified chemistry use chemistry NC"),
but the classifier doesn't classify at all, or doesn't classify into the right catefgory (not
biology or chemisty), then I would want the user to classify it manually.  So I would persist
that document and engine state, notify the user, who would classify it, and then restart the
engine, which would then move on to run the entity extractor with an NC based on the user's

Not knowing in advance where in the engine the failure will occur (failure to classify being
only one possibility), I can't create two explicit engines.  Having a general mechanism to
persist the state of the engine would let me handle any failure or missing dependency.  NLP
being generally an imprecise process, I foresee human intervention in the pipeline as a not-infrequent
occurance.  So having a mechanism to deal with that in a general way would be helpful.

This is not a high priority enhancement for me at the moment, just an idea for us to kick

Greg Holmberg

 -------------- Original message ----------------------
From: "Pascal Coupet" <pascal.coupet@temis.com>
> Hi Thomas,
> I think a way to do it is to split this process across 2 workflows. The first 
> consumer will get the CAS, eventually store it in XML somewhere (file, database 
> ...). A small application will manage the interaction with the user (sending 
> mail, reminders ...), watch a return address mailbox, update the XCAS and make 
> it available. The source of the second workflow will watch for available updated 
> XCAS and continue from there. You can in theory make the consumer of the first 
> workflow to send the mail and the source of the second watch for incoming emails 
> but it will be more difficult I think to manage properly the interaction with 
> users (reminder to responds, statistics, routing configuration ...). 
> Just some thoughts,  
> Pascal
> ________________________________
> From: Thomas Francart [mailto:thomas.francart@mondeca.com] 
> Sent: Thursday, October 11, 2007 7:01 AM
> To: uima-user@incubator.apache.org
> Subject: Asynchronous UIMA (workflow) ?
> Hi all -
> I'm thinking about whether or not it would be possible to add an asynchronous 
> step in a UIMA pipeline ? For example having an analysis engine that would ask 
> for a user input or a user review of a CAS, or something like that. Well my 
> point is that at some point in the pipeline, I would like a user to review the 
> state of the CAS, maybe add some more information, delete some others, and so 
> on; and then the rest of the pipeline would continue upon user validation. (by 
> "user" here I don't mean someone that sits in front of a computer and watch the 
> UIMA processing taking place, but maybe someone receiving an email saying "hey, 
> you should have a look and validate that").
> I know this a generic workflow question, but I was just wondering if some other 
> people had the same question/requirements with a UIMA integration, and if you 
> had some ideas on how it could be adressed/solved.
> Best,
> Thomas
> -- 
> Thomas Francart 
> Mondeca 
> 3, cité Nollez 75018 Paris France 
> Tel: +33 (0)1 44 92 35 04 - Fax: +33 (0)1 44 92 02 59 
> Blog: mondeca.wordpress.com 
> Web: www.mondeca.com 
> Mail: thomas.francart@mondeca.com 

View raw message