taverna-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stian Soiland-Reyes <soiland-re...@cs.manchester.ac.uk>
Subject Re: Taverna and SGE
Date Fri, 28 Nov 2014 01:41:41 GMT
We would welcome any effort in adding an SGE integration. Please join
the developer list if you want to discuss this in detail.
http://mail-archives.apache.org/mod_mbox/taverna-dev/

There's already interest in generalizing a "Execute this step
elsewhere" mechanism (not just for Tool activity) - so if we think of
it like this rather than just for SGE then it might get more interest
(where say SGE would just be a configuration that said "submit=qsub %1
--cluster=%2"  or something).


There are some intermediary solutions as well - although technically
Taverna would be pulling from SGE, the ENGINE and workFLOW is not -
the Tool Activity would be a single step in the workflow, and the
pulling is just part of its configuration.

Now your users might not all want to type all this qsub and so on for
every executable they want to run - so there are two faster solutions
I can think of:


a) Describe the possible executables (e.g. blast) using the "Usecase"
XML that can be imported into "Available Services". Inside the
description here, the commands could all include the
submit-pull-retrieve logic. From the Available Services users can then
drag-drop commands into the workflow and won't see the qsub etc.
unless they need to customize the parameters.
http://dev.mygrid.org.uk/wiki/display/tav250/Importing+a+set+of+tool+descriptions

b) Create External Tool activities pr executable, then package each as
a Component. These can then be shared on myExperiment as a
group/"family". Users can add this group of components to the
workbench for selection and drag-drop into the workflow. Users will
however no longer be able to easily customize the command line as
components are "gray boxes".
http://dev.mygrid.org.uk/wiki/display/tav250/Component+services



On 26 November 2014 at 18:50, Hoeftberger, Johann
<Johann_Hoeftberger@dfci.harvard.edu> wrote:
> On 11/26/2014 10:56 AM, alaninmcr wrote:
>> On 26/11/2014 15:14, Hoeftberger, Johann wrote:
>>> On 11/26/2014 08:55 AM, alaninmcr wrote:
>>>> On 25/11/2014 16:36, Hoeftberger, Johann wrote:
>>>>> Can Taverna make use of a Sun Grid Engine Cluster and is there a good
>>>>> explanation how to do that?
>>>>
>>>> What do you mean by "make use of" ? Do you want to have a workflow be
>>>> run on a node in the cluster or to have individual steps within a
>>>> workflow run be executed on nodes in the cluster?
>>>
>>> I think both.
>>> The most important need for me seems to be to have the ability that
>>> individual steps within a workflow run on our SGE cluster transparently
>>> for the user which uses the workflow. So those workflow steps should be
>>> executed on nodes of the SGE cluster, the whole handshake for the usage
>>> of those steps should be done by the Taverna system itself.
>>
>> What is the normal interface for "talking" to the SGE?
>
> That's the question whose answer I hoped the SGE integration in Taverna
> would give.
> (For our current jobs we use qsub and related tools.)
>
> It seems currently there doesn't exist a real integration of a SGE Grid
> Engine in Taverna. And the only available solution is over Tool Service,
> SSH and qsub commands.
> Although I don't have any experience with that I see the theoretical
> approach behind it. But this results in a polling strategy of the Grid
> results from a workflow engine perspectice. I thought / hoped there
> exists a tighter integrated solution for the connection between Taverna
> and SGE.
>
>
>> A quick way to try things would be use the tool service to run qsub (and
>> related) commands to your SGE. Then you could group the choreography of
>> submit -> poll * -> retrieve results, into a component.
>
> Yes, that's the best I could found and you confirm that somehow.
>
>
>> Alternatively, if there is a Java library, you could write an equivalent
>> Beanshell.
>
> I also find very interesting the "Tool Activity pluggable execution
> point" Alan mentioned in his last post. But this would take a much
> bigger development approach I guess. I have to figure out what fits best
> in our situation.
>
> Thank you all for your suggestions so far.
>
>
> Regards,
> Johann
>
>
> --
>
>
> The information in this e-mail is intended only for the person to whom it is
> addressed. If you believe this e-mail was sent to you in error and the e-mail
> contains patient information, please contact the Partners Compliance HelpLine at
> http://www.partners.org/complianceline . If the e-mail was sent to you in error
> but does not contain patient information, please contact the sender and properly
> dispose of the e-mail.
>



-- 
Stian Soiland-Reyes, myGrid team
School of Computer Science
The University of Manchester
http://soiland-reyes.com/stian/work/ http://orcid.org/0000-0001-9842-9718

Mime
View raw message