airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Heshan Suriyaarachchi <heshan.suriyaarach...@gmail.com>
Subject Re: Clarification about Airavata GFac
Date Wed, 05 Mar 2014 17:34:22 GMT
Hello Nipun,

Please see my inline comments.


On Tue, Mar 4, 2014 at 10:32 AM, Nipun Udara <y.b.n.udara@gmail.com> wrote:

> Hi all,
>
> I was trying to understand GFac module and found different providers which
> use to execute jobs like BESProvider and GRAMProvider which extends
> AbstractProvider class like to know the use of those.
>
> And when search abouth EC2 instances there are three different kind of
> instances such as reserved instances ,spot instances. i would like to know
> out of those which kind is using in GFac-ec2 module.
>

As you might know, Amazon provides a java-sdk to communicate with it's AWS.
EC2Provdier uses it to interact with AMIs and instances. When testing out
or running a workflow, I usually used a general purpose instance as I did
not have any requirement for running a workflow in compute optimized
instance family. Having said that, you will be able to run a workflow in
any kind of instance that you want. You might have to tweak the UI a bit to
support this (ie. like adding an option to a drop down, etc and modifying
the backend a bit).

As for the sample workouts that you might be planning to run, you don't
need to run them in compute optimized instances as your workflow might not
be using the full potential of the resources. Keep in mind you are paying
for the instance type as well. As you reserve an instance with higher
compute power, you might be paying more (You might want to check the
pricing of instances before running sample workflows in compute optimized
instances).

So, the short answer to your question might be, you can use which ever
instance type you want as long as the AWS java-sdk api is exposing the
capability of interacting with those instances (which they do).

And is it need to configure to pass user-data scripts that runs the job
> each time launch a job. I would be grateful if somebody can clarify
> regarding these.
>

Let me tell you how the EC2 job submissions work in Airavata.

First you should have a base image which has your application installed.
This application can be a jar/shell script or other kind of a script. Try
and wrap your application with a shell script and make it available for
shell execution.

The EC2 Provider will get the user information (secret key, access key, AMI
id) and start up an instance from the base image (which has your
application installed.) Once the instance is running, the EC2 provider
establishes a SSH connection to the running instance and then runs the
shell script (which wraps your application) with the user given input
params. Provider will poll the standard out from the script and extract the
result out. Then it will be returned to the provider. Then this will passed
on to the workflow.

AFAICR I used J2ssh for the ssh connections. Since you are getting the
shell access to the instance, you might be able to run any script type that
you want but please remember that if you are using any script that is
dependent on the environment/version, you might run into some issues (eg.
Running a python 3.0 script on a Python 2.5 system). These things are also
minor issues and can be easily overcome. The reason why I told you this is
that, then you would not get side tracked because of any environment
issues.

I didn't check Airavata mailing list for a long time. Therefore, didn't see
this. :) Lahiru, pointed me to this thread. If you have any
questions/doubts, please don't hesitate to ask.

Thanks,
Heshan.

>
> Regards
> Nipun Udara
>
> *Undergraduate*
> *Department of Computer Science & Engineering*
> *University of Moratuwa*
> *Sri Lanka*
>



-- 
Regards,
Heshan Suriyaarachchi

http://heshans.blogspot.com/
http://www.linkedin.com/in/heshan

Mime
View raw message