taverna-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marlon Pierce <marpi...@iu.edu>
Subject Re: Taverna and SGE
Date Fri, 28 Nov 2014 15:23:33 GMT
Now is an excellent time to get involved, with Taverna going through 
Apache incubation.  Demonstrating community contributions and welcoming 
new committers is part of the process.


On 11/27/14, 8:41 PM, Stian Soiland-Reyes wrote:
> We would welcome any effort in adding an SGE integration. Please join
> the developer list if you want to discuss this in detail.
> http://mail-archives.apache.org/mod_mbox/taverna-dev/
> There's already interest in generalizing a "Execute this step
> elsewhere" mechanism (not just for Tool activity) - so if we think of
> it like this rather than just for SGE then it might get more interest
> (where say SGE would just be a configuration that said "submit=qsub %1
> --cluster=%2"  or something).
> There are some intermediary solutions as well - although technically
> Taverna would be pulling from SGE, the ENGINE and workFLOW is not -
> the Tool Activity would be a single step in the workflow, and the
> pulling is just part of its configuration.
> Now your users might not all want to type all this qsub and so on for
> every executable they want to run - so there are two faster solutions
> I can think of:
> a) Describe the possible executables (e.g. blast) using the "Usecase"
> XML that can be imported into "Available Services". Inside the
> description here, the commands could all include the
> submit-pull-retrieve logic. From the Available Services users can then
> drag-drop commands into the workflow and won't see the qsub etc.
> unless they need to customize the parameters.
> http://dev.mygrid.org.uk/wiki/display/tav250/Importing+a+set+of+tool+descriptions
> b) Create External Tool activities pr executable, then package each as
> a Component. These can then be shared on myExperiment as a
> group/"family". Users can add this group of components to the
> workbench for selection and drag-drop into the workflow. Users will
> however no longer be able to easily customize the command line as
> components are "gray boxes".
> http://dev.mygrid.org.uk/wiki/display/tav250/Component+services
> On 26 November 2014 at 18:50, Hoeftberger, Johann
> <Johann_Hoeftberger@dfci.harvard.edu> wrote:
>> On 11/26/2014 10:56 AM, alaninmcr wrote:
>>> On 26/11/2014 15:14, Hoeftberger, Johann wrote:
>>>> On 11/26/2014 08:55 AM, alaninmcr wrote:
>>>>> On 25/11/2014 16:36, Hoeftberger, Johann wrote:
>>>>>> Can Taverna make use of a Sun Grid Engine Cluster and is there a
>>>>>> explanation how to do that?
>>>>> What do you mean by "make use of" ? Do you want to have a workflow be
>>>>> run on a node in the cluster or to have individual steps within a
>>>>> workflow run be executed on nodes in the cluster?
>>>> I think both.
>>>> The most important need for me seems to be to have the ability that
>>>> individual steps within a workflow run on our SGE cluster transparently
>>>> for the user which uses the workflow. So those workflow steps should be
>>>> executed on nodes of the SGE cluster, the whole handshake for the usage
>>>> of those steps should be done by the Taverna system itself.
>>> What is the normal interface for "talking" to the SGE?
>> That's the question whose answer I hoped the SGE integration in Taverna
>> would give.
>> (For our current jobs we use qsub and related tools.)
>> It seems currently there doesn't exist a real integration of a SGE Grid
>> Engine in Taverna. And the only available solution is over Tool Service,
>> SSH and qsub commands.
>> Although I don't have any experience with that I see the theoretical
>> approach behind it. But this results in a polling strategy of the Grid
>> results from a workflow engine perspectice. I thought / hoped there
>> exists a tighter integrated solution for the connection between Taverna
>> and SGE.
>>> A quick way to try things would be use the tool service to run qsub (and
>>> related) commands to your SGE. Then you could group the choreography of
>>> submit -> poll * -> retrieve results, into a component.
>> Yes, that's the best I could found and you confirm that somehow.
>>> Alternatively, if there is a Java library, you could write an equivalent
>>> Beanshell.
>> I also find very interesting the "Tool Activity pluggable execution
>> point" Alan mentioned in his last post. But this would take a much
>> bigger development approach I guess. I have to figure out what fits best
>> in our situation.
>> Thank you all for your suggestions so far.
>> Regards,
>> Johann
>> --
>> The information in this e-mail is intended only for the person to whom it is
>> addressed. If you believe this e-mail was sent to you in error and the e-mail
>> contains patient information, please contact the Partners Compliance HelpLine at
>> http://www.partners.org/complianceline . If the e-mail was sent to you in error
>> but does not contain patient information, please contact the sender and properly
>> dispose of the e-mail.

View raw message