Return-Path: X-Original-To: apmail-airavata-dev-archive@www.apache.org Delivered-To: apmail-airavata-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8BFB119719 for ; Mon, 21 Mar 2016 14:23:14 +0000 (UTC) Received: (qmail 27783 invoked by uid 500); 21 Mar 2016 14:23:14 -0000 Delivered-To: apmail-airavata-dev-archive@airavata.apache.org Received: (qmail 27733 invoked by uid 500); 21 Mar 2016 14:23:14 -0000 Mailing-List: contact dev-help@airavata.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airavata.apache.org Delivered-To: mailing list dev@airavata.apache.org Received: (qmail 27723 invoked by uid 99); 21 Mar 2016 14:23:14 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2016 14:23:14 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id B6FC518021D for ; Mon, 21 Mar 2016 14:23:13 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.989 X-Spam-Level: * X-Spam-Status: No, score=1.989 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_KAM_HTML_FONT_INVALID=0.01] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=binghamton-edu.20150623.gappssmtp.com Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id z6iKmibEGUhU for ; Mon, 21 Mar 2016 14:23:11 +0000 (UTC) Received: from mail-yw0-f169.google.com (mail-yw0-f169.google.com [209.85.161.169]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with ESMTPS id 9AA2A5F246 for ; Mon, 21 Mar 2016 14:23:11 +0000 (UTC) Received: by mail-yw0-f169.google.com with SMTP id h129so216439241ywb.1 for ; Mon, 21 Mar 2016 07:23:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=binghamton-edu.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to; bh=Y3Yz1YGqWNdNha/cfUiAgzf1BCo+wfv5wl1eTubXIJg=; b=qkSM+0lpzge9WnESeuz/3Uqn/qaaVwnEHXycnMsr6XOpWnsyHVc+y7Mw33OXRjgngB RfhfcunEb8OVtlstGZmqrx+ejlXWHI7mdNbWsPxFpC58/8+0iILDoyInVz08gylNV7Kt 74+X2qS7VuTDnLE7E4TappMF3AIAVwFcVoKjoQae5F9RUrl62Q/3f52Q2YSfbL/sPS8d VwBQgFxbkDiiWHi2AgA5aGxys09GEUOH3iEPJVbh9Gdco7YqgaEA8KxmrbFFSXu+Z/DT /2HjoTa340HbyLAzCME1i3zvjt7C4K8wm273lDnZ/zRk31bu/wmzbyHWLMZ64Z/+uCUo AaHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=Y3Yz1YGqWNdNha/cfUiAgzf1BCo+wfv5wl1eTubXIJg=; b=CW12FFYb3T0bnEKsGckS+mj3hHSuCUUKxrYxSUM9Sw1UeHShh26TeLEBzg69E89N0w KxbnuTapsY2JnEERI5Qh59cciBz3zhnrhDy6MJT3jJbHiZb6J52Belrf3ysSYmWjvBrh jAvKH7Vox+fNeO7LAtYJ7MncBrSy3QnOwoxcWUGFRl7llLmpJAa6BTuk9jtsh4TB/6xI POiuyY5ZovPEMynhiCXNCeKAxXrFEevtxqLub5UySCjg23IYe6F93hKfOwZtYZiOT2Xf XpkFNJxqNurEVodcaCeKkSYavKGvHPIsi12s/t2ygU3Awhg0lhxn5QJp+g+CZBnqRo1u nr7w== X-Gm-Message-State: AD7BkJI5QOHxQiOlLQjWBmAH8pMh5+Xi6r8JxmR4e4O5gneug4/Ow5j+hZfYJeI+hIqNPjW9uPgrayQlH9UejB9n X-Received: by 10.37.101.138 with SMTP id z132mr7588236ybb.2.1458570190946; Mon, 21 Mar 2016 07:23:10 -0700 (PDT) MIME-Version: 1.0 Received: by 10.37.92.84 with HTTP; Mon, 21 Mar 2016 07:22:31 -0700 (PDT) From: Gourav Rattihalli Date: Mon, 21 Mar 2016 10:22:31 -0400 Message-ID: Subject: [GSoC Proposal] - Integrating Job and Cloud Health Information of Apache Aurora with Apache Airavata To: dev@airavata.apache.org Content-Type: multipart/alternative; boundary=001a1142fd801cb4ba052e8fd697 --001a1142fd801cb4ba052e8fd697 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Dev Team, Please review the following GSoC proposal that I plan to submit: *Title*: Integrating Job and Cloud Health Information of Apache Aurora with Apache Airavata *Abstract*: This project will incorporate Apache Aurora to enable Airavata to launch jobs on large cloud environments, and collect the related information on the health of each job and the cloud resources. The project will also analyze the current micro-services architecture of Airavata and develop code for an updated architecture for modules such as Logging. As as result, another outcome of this project would be development of a module that will collect all the logging information from the various execution points in an Airavata job's lifecycle and provide search and mining capability. *Introduction*: Apache Aurora is a service scheduler, that runs on top of Apache Mesos. This combination enables the use of long running services that take advantage of Apache Mesos scalability, fault-tolerance and resource isolation. Apache Mesos is a cluster manager, which provides information about the state of the cluster. Aurora uses that knowledge to make scheduling decisions. For example, when a machine experiences failure Aurora automatically reschedules those previously-running services onto a healthy machine in order to keep them running. Each job is tracked by Aurora to be in one of the following states: pending, assigned, starting, running, and finished. Apache Aurora requires a configuration file =E2=80=9D.aurora=E2=80=9D to la= unch jobs. Following is an example of Aurora configuration file: import os hello_world_process =3D Process(name =3D 'hello_world', cmdline =3D 'echo h= ello world') hello_world_task =3D Task( resources =3D Resources(cpu =3D 0.1, ram =3D 16 * MB, disk =3D 16 * MB), processes =3D [hello_world_process]) hello_world_job =3D Job( cluster =3D 'cluster1', role =3D os.getenv('USER'), task =3D hello_world_task) jobs =3D [hello_world_job] To launch the job with the above configuration we use: aurora job create cluster1/$USER/test/hello_world hello_world.aurora This project will develop modules in Airavata to automatically generate the Aurora configuration file to launch a job on an Aurora-managed cluster in a cloud environment. The Aurora user interface, as shown in the web portal displayed above, provides detailed information on the job status, job name, start and finish times, location of the logs, and resource usage. This project will use add a module to Apache Aurora to pull this detailed information using the the Aurora HTTP API. *Goals*: - This project will investigate how apache Aurora collects information of cluster environment for display on the Aurora web interface. We will stu= dy the Aurora HTTP API and retrieve all the information related to the targ= et infrastructure and job health, and make it available to the Airavata job submission module. - We will process the retrieved information from Aurora and convert the information in a format that can be used by Airavata for further action. - We will use the appropriate design patterns to integrate the use of Aurora as one of the options for Big Data and Cloud resource frameworks with the Airavata framework - We will make the resource information from Aurora available for display on the Airavata dashboard. Any comment and suggestions would be very helpful. -Gourav Rattihalli --001a1142fd801cb4ba052e8fd697 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Dev Team,

Please review the following GSoC proposal that I plan to subm= it:

Title:=C2=A0Integrating= Job and Cloud Health Information of Apache Aurora with Apache Airavata

Abstract:=C2=A0

This project will incorporate Apache Aurora to enable Airavata to launch jobs on large cloud environments, and collect the related information on the health= of each job and the cloud resources. The project will also analyze the current micro-services architecture of Airavata and develop code for an updated architecture for modules such as Logging. As as result, another outcome of = this project would be development of a module that will collect all the logging information from the various execution points in an Airavata job's life= cycle and provide search and mining capability.

Introduction:=C2=A0

Apache Aurora is a service scheduler, that runs on top of Apach= e Mesos. This combination enables the use of long running services that tak= e advantage of Apache Mesos scalability, fault-tolerance and resource isola= tion. Apache = Mesos is a cluster manager, which provides information about the state of t= he cluster. Aurora uses that knowledge to make scheduling decisions. For ex= ample, when a machine experiences failure Aurora automatically reschedules = those previously-running services onto a healthy machine in order to keep t= hem running. Each job is tracked by Aurora to be in one of the following st= ates: pending, assigned, starting, running, and finished.


Apache Auror= a requires a configuration file =E2=80=9D.aurora=E2=80=9D to launch jobs. F= ollowing is an example of Aurora configuration file:


import os<= /span>
hello_world_process =3D= Process(name =3D 'hello_world', cmdline =3D 'echo hello world&= #39;)

hello_world_task =3D Task(=
=C2=A0resources =3D R= esources(cpu =3D 0.1, ram =3D 16 * MB, disk =3D 16 * MB),
 =C2=A0processes =3D [hello_world_proces= s])

hello_world_job =3D Job(
 =C2=A0cluster =3D 'cl= uster1',<= br> =C2=A0rol= e =3D os.getenv('USER'),
 =C2=A0task =3D hello_world_task)

jobs =3D [hello_world_job]


To launch the job with the ab= ove configuration we use:


aurora job create cluster1/$USER/test/= hello_world hello_world.aurora


This project will de= velop modules in Airavata to automatically generate the Aurora configuratio= n file to launch a job on an Aurora-managed cluster in a cloud environment.= The Aurora user interface, as shown in the web portal displayed above, pro= vides detailed information on the job status, job name, start and finish ti= mes, location of the logs, and resource usage. This project will use add a = module to Apache Aurora to pull this detailed information using the the Aur= ora HTTP API.


Goals:=C2=A0

  • This project will investigate how apache Aurora collects information of= cluster environment for display on the Aurora web interface. We will study= the Aurora HTTP API and retrieve all the information related to the target= infrastructure and job health, and make it available to the Airavata job s= ubmission module.

  • We wil= l process the retrieved information from Aurora and convert the information= in a format that can be used by Airavata for further action.

  • We will use the appropriate design pat= terns to integrate the use of Aurora as one of the options for Big Data and= Cloud resource frameworks with the Airavata framework

  • We will make the resource information from Au= rora available for display on the Airavata dashboard. =C2=A0


Any c= omment and suggestions would be very helpful.

-G= ourav Rattihalli
--001a1142fd801cb4ba052e8fd697--