uima-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John David Osborne <ozb...@uab.edu>
Subject Re: Running UIMA on a cluster
Date Fri, 27 Apr 2012 21:48:04 GMT
Very helpful responses from you and Thomas, thanks guys!  The README in
the 2.3.1 documentation is very useful.

I'm still confused about one thing, and I am dreading the answer. How does
UIMA-AS play with pre-existing tools like SGE? I'm under the impression
that it is basically going to ignore SGE and try to start jobs on the
compute nodes by itself. Is everybody running UIMA on  dedicated clusters
more or less?

I'm in a situation where I'm looking to run on a cluster shared pretty
much University wide for which SGE is the main (probably only) job
submission method.

 -John



On 4/27/12 2:59 PM, "Eric Riebling" <er1k@cs.cmu.edu> wrote:

>We've had success deploying annotators on cluster nodes (using UIMA-AS
>deployment descriptors) registered to a UIMA-AS broker running on the
>head node.  If the cluster use shared data folders, you only need to
>put the code in one place for it to 'appear' on all nodes.
>
>Then we run a collection reader and CAS consumer on the head node,
>with the amount of scale-out specified on the command line of
>runRemoteAsyncAE.sh, something like this:
>
>   $UIMA_HOME/bin/runRemoteAsyncAE.sh -c (path.to)XmiCollectionReader.xml
>tcp://localhost:6
>1616 (name of deployed service) -p (number of nodes) -o output_foldername
>
>With enough scale-out, the limiting factor becomes the speed of the CR
>and CC on the head node.  This is the briefest explanation I can give,
>not sure it's a 'best practice' but it works. :)
>
>On 4/27/2012 3:35 PM, John David Osborne wrote:
>> Hello,
>>
>> Is there any best practice documentation out there for running
>> UIMA/UIMA-AS on a cluster? I have only run single machine instances of
>> UIMA (mostly through Eclipse) and have not investigated the ability to
>> perform multiple simultaneous analyses in order to process large
>>document
>> collections.
>>
>> It's not clear to me how UIMA would operate in a cluster environment, do
>> people really do message passing using JMI? I'm guessing this is the
>>case
>> as I seeing references to MPICH, SGE or other things I am more used to.
>> I've looked through some of the documentation (including all the
>>Overview
>> &  SDK setup) but am not finding anything helpful. I've also tried
>>googling
>> but I am not getting much except this:
>> http://comments.gmane.org/gmane.comp.apache.uima.general/2131 which
>>makes
>> me think it is possible.
>>
>> Currently with my level of confusion I think it may be best to have
>> multiple instances of UIMA on a cluster and just submit jobs processing
>> discrete document sets to our SGE cluster and ignore whatever scaling
>> features are actually present in UIMA since the document processing I
>>plan
>> to do is data parallel.
>>
>> -John
>>
>>
>
>-- 
>Eric Riebling                 Senior Systems Programmer
>http://ericriebling.com       CMU Language Technologies Institute
>      


Mime
View raw message