airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pankaj Saha <>
Subject [GSoC Proposal] - Integrating Resource Information from Apache Mesos with Apache Airavata’s Job Management Modules
Date Mon, 21 Mar 2016 15:16:36 GMT
Hi Dev Team,

Please review the following GSoC proposal that I plan to submit:
Title: Integrating Resource Information from Apache Mesos with Apache
Airavata’s Job Management Modules

Apache Airavata provides gateway computing capability across clustered
environments for scientific users. It abstracts away the complexities of
submitting jobs to HPC platforms and provides users with an intuitive and
elegant web-based interface to submit jobs. Apache Mesos is a  distributed
kernel that manages distributed computing resources as a single computer.
As Airavata is being extended to use Big Data and Cloud tools to launch
jobs in cloud environments, it needs to retrieve the resource and job
execution information from the Big Data framework back to the Apache portal
accessible to the end user. In this project we will develop code and
scripts to be integrated with the Airavata that will use the HTTP API of
Mesos to continuously fetch the complete resource and scheduling
information. This information can then be used by Airavata to dynamically
monitor and improve its job submission strategy in cloud environments such
as Jetstream.

Apache Mesos provides HTTP API endpoints for scheduler, executor, internal
and admin related queries. To fetch information regarding a clustered
environment that is managed by the Mesos master, the API can be accessed
via curl requests over HTTP. The response to such requests will be received
as well formed json document. We will parse the json response and present
the information in the format desired. The retrieved information will
include resource usage, resource available for further jobs, job status,
time elapsed since the job started, etc.  Airavata, in turn, will use this
information to determine the resource usage, performance of the jobs on a
job submission, rapid diagnosis on the health of the submitted jobs.

We will use the observer pattern to continuously pull information from
Cloud and big Data Resource Managers, such as Apache Mesos, to Airavata.

Any comment and suggestions would be very helpful.


View raw message