airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pierce, Marlon" <marpi...@iu.edu>
Subject Re: [GSoC Proposal] - Integrating Resource Information from Apache Mesos with Apache Airavata’s Job Management Modules
Date Wed, 23 Mar 2016 14:28:29 GMT
Hi Pankaj,

I still don’t see your application in the GSOC site. Please make sure you have done this
correctly.

Thanks,

Marlon

From: Pankaj Saha <psaha4@binghamton.edu<mailto:psaha4@binghamton.edu>>
Date: Monday, March 21, 2016 at 5:17 PM
To: marpierc <marpierc@iu.edu<mailto:marpierc@iu.edu>>
Cc: "dev@airavata.apache.org<mailto:dev@airavata.apache.org>" <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Subject: Re: [GSoC Proposal] - Integrating Resource Information from Apache Mesos with Apache
Airavata’s Job Management Modules

Hi Marlon,
Here is the link that I have created.

https://docs.google.com/document/d/1qtFvg4-usT4D_1TDNBsQDFQGZIkH99ideYQ1T3HU9nY/edit?usp=sharing
The draft is created under the GSoC proposal site under Apache foundation with the same title.


Thanks
Pankaj



On Mon, Mar 21, 2016 at 4:47 PM, Pierce, Marlon <marpierc@iu.edu<mailto:marpierc@iu.edu>>
wrote:
Hi Pankaj,

I have some comments, but it would be easier if you created a proposal draft in the GSOC site.
The google doc option for your draft is better than pointing to the Airavata wiki. Please
make sure you give comment and suggestion permissions.

Marlon


From: Pankaj Saha <psaha4@binghamton.edu<mailto:psaha4@binghamton.edu>>
Reply-To: "dev@airavata.apache.org<mailto:dev@airavata.apache.org>" <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Date: Monday, March 21, 2016 at 11:16 AM
To: dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Subject: [GSoC Proposal] - Integrating Resource Information from Apache Mesos with Apache
Airavata’s Job Management Modules

Hi Dev Team,

Please review the following GSoC proposal that I plan to submit:
Title: Integrating Resource Information from Apache Mesos with Apache Airavata’s Job Management
Modules

Abstract:
Apache Airavata provides gateway computing capability across clustered environments for scientific
users. It abstracts away the complexities of submitting jobs to HPC platforms and provides
users with an intuitive and elegant web-based interface to submit jobs. Apache Mesos is a
 distributed kernel that manages distributed computing resources as a single computer. As
Airavata is being extended to use Big Data and Cloud tools to launch jobs in cloud environments,
it needs to retrieve the resource and job execution information from the Big Data framework
back to the Apache portal accessible to the end user. In this project we will develop code
and scripts to be integrated with the Airavata that will use the HTTP API of Mesos to continuously
fetch the complete resource and scheduling information. This information can then be used
by Airavata to dynamically monitor and improve its job submission strategy in cloud environments
such as Jetstream.

Introduction:
Apache Mesos provides HTTP API endpoints for scheduler, executor, internal and admin related
queries. To fetch information regarding a clustered environment that is managed by the Mesos
master, the API can be accessed via curl requests over HTTP. The response to such requests
will be received as well formed json document. We will parse the json response and present
the information in the format desired. The retrieved information will include resource usage,
resource available for further jobs, job status, time elapsed since the job started, etc.
 Airavata, in turn, will use this information to determine the resource usage, performance
of the jobs on a job submission, rapid diagnosis on the health of the submitted jobs.

We will use the observer pattern to continuously pull information from Cloud and big Data
Resource Managers, such as Apache Mesos, to Airavata.

Any comment and suggestions would be very helpful.

Thanks
Pankaj


Mime
View raw message