airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Danushka Menikkumbura <danushka.menikkumb...@gmail.com>
Subject Re: Support high-throughput computing and development/integration strategies
Date Sun, 03 Mar 2013 03:52:51 GMT
As I understand, Airavata should already be able to invoke jobs in an HTC
environment (GramProvider). If not it should be a little variation of the
current the current provider. We should also be able to have support for
different types of job in XBaya and provision them in the providers
(new/existing).

What I am not sure about is whether different HTC providers use a unified
API. For example API of Condor may be completely different from them of OSG
and in which case Airvata would end up having different extensions for
different HTC providers.

Monitoring (provenance) HTC jobs is another concern.

Thanks,
Danushka


On Sun, Mar 3, 2013 at 9:01 AM, Raminder Singh <raminderjsingh@gmail.com>wrote:

> Amila, Condor is a middleware to provision HTC resources. Following slides
> can provide basic information.
>
> http://research.cs.wisc.edu/htcondor/tutorials/condor-g-dagman-talk.ppt
>
> Yan, If i understood it right, collecting batch job and parameter-sweep
> information to construct a DAG is a good idea and its a good use-case for
> airavata.  Its going to be interesting for managing large number of jobs
> running in parallel.
>
> Thanks
> Raminder
>
> On Mar 2, 2013, at 9:56 PM, Amila Jayasekara wrote:
>
> > Hi Yan,
> >
> > This is a great idea. Please some inline comments below.
> >
> > Thanks
> > Amila
> >
> > On Sat, Mar 2, 2013 at 5:43 PM, Yan Liu <yanliu@cybergis.org> wrote:
> >> High-throughput computing (HTC) resources are available from national
> >> cyberinfrastructures such as the Open Science Grid and NSF XSEDE. HTC
> >> resources are suitable for running serial or embarrassingly parallel
> user
> >> jobs. Unlike high-performance computing (HPC) resources, HTC
> environment is
> >> more distributed, loosely coupled, and managed individually by various
> >> resource contributors. For HTC users, a HTC environment presents a
> virtual
> >> resource pool with dynamically aggregated resources and can be accessed
> >> through a unified client software (e.g., OSG/VDT/Condor).
> >>
> >> Integrating HTC capabilities in Airavata is important for users to
> access
> >> HTC resources seamlessly as they access other kinds of computing
> >> environments supported by Airavata. At first glance, the integration
> may be
> >> straightforward by adding middleware support (e.g., Condor or BOSCO)
> into
> >> Airavata. However, I am proposing a user-oriented approach to the
> >> integration in order to fully leverage HTC client software's
> capabilities.
> >>
> >> An Airavata user does not care the underlying middleware when she/he
> >> composes a job, ideally. What the user cares is the computational
> capability
> >> provided by the underlying resources. A HTC environment, with the
> support
> >> from the Condor middleware, is desirable for running:
> >> - large batch jobs
> >> - parameter-sweeping jobs
> >> - stochastic jobs with the same configuration but requiring a large
> number
> >> of repeated runs in order to obtain statistically confident results
> >> - workflow jobs that can be represented as DAG (directed acyclic graph)
> >
> > I am +1 for this idea. So how do you precisely define a HTC
> > environment ? (In other words what parameters does user needs to
> > specify when configuring a HTC environment ?)
> > I was kind of thinking along the same path. I agree with you that user
> > does not need to care about underlying middleware where his/her job
> > should run. So Airavata should be able to figure out an appropriate
> > place (machine) to run the job based on provided configurations.  I
> > think that configuration should be what you proposed as "HTC
> > environment".
> >
> > I am also curious to know whether you have thought about how
> > middleware related security (authentication, proxy certificates etc
> > ...) should be handle in this scenario. (My knowledge about Condor is
> > limited)
> >
> > Further I didnt quite understand how Condor middleware enables you to
> > run "workflow jobs that can be represented as DAG". Appreciate your
> > explanation on this.
> >
> > Also can Condor work with providers such as EC2 ?
> >
> >>
> >> Therefore, instead of presenting a raw Condor interface to Airavata
> users,
> >> tailored interfaces to aforementioned user job types will be more
> useful.
> >> Technically, Condor submmit script syntax supports all of the described
> jobs
> >> through job macros and DAG support. If Airavata can bridge user job
> >> requirements and the composition of the technical Condor submission
> script,
> >> HTC resources can be more effectively represented for and used by
> Airavata
> >> community.
> >>
> >> The development roadmap is upon Airavata team's design, I'm willing to
> >> contribute a disease mapping application for the testing and evaluation
> of
> >> the new components and capabilities developed in Airavata for this
> purpose.
> >>
> >> Thanks,
> >> Yan
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message