Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 8751F200BAE for ; Fri, 28 Oct 2016 18:47:08 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 85E53160AE4; Fri, 28 Oct 2016 16:47:08 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 81DCB160ACA for ; Fri, 28 Oct 2016 18:47:07 +0200 (CEST) Received: (qmail 84595 invoked by uid 500); 28 Oct 2016 16:47:06 -0000 Mailing-List: contact dev-help@airavata.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airavata.apache.org Delivered-To: mailing list dev@airavata.apache.org Received: (qmail 84581 invoked by uid 99); 28 Oct 2016 16:47:06 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Oct 2016 16:47:06 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 08BDC180601 for ; Fri, 28 Oct 2016 16:47:06 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.379 X-Spam-Level: ** X-Spam-Status: No, score=2.379 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id jdrEBe4EAkLi for ; Fri, 28 Oct 2016 16:47:02 +0000 (UTC) Received: from mail-wm0-f53.google.com (mail-wm0-f53.google.com [74.125.82.53]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id C66355F54E for ; Fri, 28 Oct 2016 16:47:01 +0000 (UTC) Received: by mail-wm0-f53.google.com with SMTP id p190so23273445wmp.1 for ; Fri, 28 Oct 2016 09:47:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=mL6fx4KacZCmghvQK9Dgf+AGSDSeanfc0UmszRHfucc=; b=tv5OazMBS7R4N18deom9iWd3ERAHydhIJrmCuVuI9tE0Y9jvMkQVhm5PTxDdsL555n agHLGVnK8KhFZJMsvlZP+kxIxkPYsoe8L0KxcrBjy/Iua4nLvyorTagRdqFThMRzuMrz Ub0EWE7Pvka01bCZzc9ygPqxU6XweoC/KzX8uONmI8lE74ohagHXhsRMch2w5uk+vVNK Bfrsfi8YTKelKjCjWerGd2gUj154a7UHCWiPn3TrB7ciDsBC3mdb9j65RYdcSqsUVEzR aN1RPaRCKjAb+uXRF1dhSJwWcAFe+zwrRS2pQF1Am5o/yZ7+lC2hQ1usjl5FDHAjnn3k 2gAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=mL6fx4KacZCmghvQK9Dgf+AGSDSeanfc0UmszRHfucc=; b=F/v+RAOkfQRroy6Er39ceZhNAJzE4UTgO58/g+1uS30aiNq7aA/TnB8+03YLHpxRIK CT7nLtliOGynkici3NIfJWSttpebN9D8Yf/MQHr0qH0jWIU5V3n4Yv4H1nSN/xV0Ciu0 To6L8bwBq+ssHKqz6Z1rU0C8hVdtDRsm5wh6Mfv8iApqGtaOjsGwcs8EYCYPcFfcp5r3 dvAOjnaCM+R3MHkiERz6cleRxClzJGVPZ21spKWZAbs+jmdxBuReenOG325H1+XnqUMW z4BVWXJHqFQvdLV3+v2eRZG+vHiMYaZhlkDerA0Co/r17koNjO20KOzY9BNjqU9ryGvW HpRQ== X-Gm-Message-State: ABUngvc0GhRgw4P4OfES1p6jiiM6o0O/0ZcRrRVvoOvKTnkWypDv4K1Cxb6q2OZ5aXHQOr9I6S6qRh7NaTi5kw== X-Received: by 10.28.66.26 with SMTP id p26mr4288220wma.120.1477673218510; Fri, 28 Oct 2016 09:46:58 -0700 (PDT) MIME-Version: 1.0 Received: by 10.80.136.85 with HTTP; Fri, 28 Oct 2016 09:46:58 -0700 (PDT) In-Reply-To: References: From: Mangirish Wagle Date: Fri, 28 Oct 2016 12:46:58 -0400 Message-ID: Subject: Re: mesos and moving jobs between clusters To: dev@airavata.apache.org Content-Type: multipart/alternative; boundary=001a114bda32489434053fef9b0e archived-at: Fri, 28 Oct 2016 16:47:08 -0000 --001a114bda32489434053fef9b0e Content-Type: text/plain; charset=UTF-8 Hi Pankaj, I was curious to know what is your motivation to work on developing a custom framework and not use Aurora or any existing robust frameworks. It would be great if you could share some pointers on that. I would also like to know what specific use cases you are targeting through your framework, as well as what are various stability concerns that you may have identified and how are you planning to handle them? Regards, Mangirish On Tue, Oct 25, 2016 at 6:09 PM, Pankaj Saha wrote: > Hi Mark, > > Mesos collects the resource information from all the nodes in the cluster > (cores, memory, disk, and gpu) and presents a unified view, as if it is a > single operating system. The Mesosphere, who a commercial entity for Mesos, > has built an ecosystem around Mesos as the kernel called the "Data Center > Operating System (DCOS)". Frameworks interact with Mesos to reserve > resources and then use these resources to run jobs on the cluster. So, for > example, if multiple frameworks such as Marathon, Apache Aurora, and a > custom-MPI-framework are using Mesos, then there is a negotiation between > Mesos and each framework on how many resources each framework gets. Once > the framework, say Aurora, gets resources, it can decide how to use those > resources. Some of the strengths of Mesos include fault tolerance at scale > and the ability to co-schedule applications/frameworks on the cluster such > that cluster utilization is high. > > Mesos off-the-shelf only works when the Mater and agent nodes have a line > of communication to each other. We have worked on modifying the Mesos > installation so that it even works when agents are behind firewalls on > campus clusters. We are also working on getting the same setup to work on > Jetstream and Chameleon where allocations are a mix of public IPs and > internally accessible nodes. This will allow us to use Mesos to > meta-schedule across clusters. We are also developing our own framework, to > be able to customize scheduling and resource negotiations for science > gateways on Mesos clusters. Our plan is to work with Suresh and Marlon's > team so that it works with Airavata. > > I will be presenting at the Gateways workshop in November, and then I will > also be at SC along with my adviser (Madhu Govindaraju), if you would like > to discuss any of these projects. > > We are working on packaging our work so that it can be shared with this > community. > > > Thanks > > Pankaj > > On Tue, Oct 25, 2016 at 11:36 AM, Mangirish Wagle < > vaglomangirish@gmail.com> wrote: > >> Hi Mark, >> >> Thanks for your question. So if I understand you correctly, you need kind >> of load balancing between identical clusters through a single Mesos master? >> >> With the current setup, from what I understand, we have a separate mesos >> masters for every cluster on separate clouds. However, its a good >> investigative topic if we can have single mesos master targeting multiple >> identical clusters. We have some work ongoing to use a virtual cluster >> setup with compute resources across clouds to install mesos, but not sure >> if that is what you are looking for. >> >> Regards, >> Mangirish >> >> >> >> >> On Tue, Oct 25, 2016 at 11:05 AM, Miller, Mark wrote: >> >>> Hi all, >>> >>> >>> >>> I posed a question to Suresh (see below), and he asked me to put this >>> question on the dev list. >>> >>> So here it is. I will be grateful for any comments about the issues you >>> all are facing, and what has come up in trying this, as >>> >>> It seems likely that this is a much simpler problem in concept than it >>> is in practice, but its solution has many benefits. >>> >>> >>> >>> Here is my question: >>> >>> A group of us have been discussing how we might simplify submitting jobs >>> to different compute resources in our current implementation of CIPRES, and >>> how cloud computing might facilitate this. But none of us are cloud >>> experts. As I understand it, the mesos cluster that I have been seeing in >>> the Airavata email threads is intended to make it possible to deploy jobs >>> to multiple virtual clusters. I am (we are) wondering if Mesos manages >>> submissions to identical virtual clusters on multiple machines, and if that >>> works efficiently. >>> >>> >>> >>> In our implementation, we have to change the rules to run efficiently on >>> different machines, according to gpu availability, and cores per node. I am >>> wondering how Mesos/ virtual clusters affect those considerations. >>> >>> Can mesos create basically identical virtual clusters independent of >>> machine? >>> >>> >>> Thanks for any advice. >>> >>> >>> >>> Mark >>> >>> >>> >>> >>> >>> >>> >>> >>> >> >> > --001a114bda32489434053fef9b0e Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Pankaj,

I was curious to know what i= s your motivation to work on developing a custom framework and not use Auro= ra or any existing robust frameworks. It would be great if you could share = some pointers on that.
I would also like to know what specific us= e cases you are targeting through your framework, as well as what are vario= us stability concerns that you may have identified and how are you planning= to handle them?

Regards,
Mangirish




On Tue, Oct 25, 2016 at 6:09 PM, Pankaj Saha <psaha4@bing= hamton.edu> wrote:

Hi Mark,=


Mesos collects the resource information from all the= nodes in the cluster (cores, memory, disk, and gpu) and presents a unified= view, as if it is a single operating system. The Mesosphere, who a commerc= ial entity for Mesos, has built an ecosystem around Mesos as the kernel cal= led the "Data Center Operating System (DCOS)".=C2=A0 Frameworks i= nteract =C2=A0with Mesos to reserve resources and then use these resources = to run jobs on the cluster. So, for example, if multiple frameworks such as= Marathon, Apache Aurora, and a custom-MPI-framework are using Mesos, then = there is a negotiation between Mesos and each framework on how many resourc= es each framework gets. Once the framework, say Aurora, gets resources, it = can decide how to use those resources. Some of the strengths of Mesos inclu= de fault tolerance at scale and the ability to co-schedule applications/fra= meworks on the cluster such that cluster utilization is high.


Mesos off-the-shelf only works when the Mater and agent nodes have= a line of communication to each other. We have worked on modifying the Mes= os installation so that it even works when agents are behind firewalls on c= ampus clusters. We are also working on getting the same setup to work on Je= tstream and Chameleon where allocations are a mix of public IPs and interna= lly accessible nodes. This will allow us to use Mesos to meta-schedule acro= ss clusters. We are also developing our own framework, to be able to custom= ize scheduling and resource negotiations for science gateways on Mesos clus= ters. Our plan is to work with Suresh and Marlon's team so that it work= s with Airavata. =C2=A0


I will be presenting at the G= ateways workshop in November, and then I will also be at SC along with my a= dviser (Madhu Govindaraju), if you would like to discuss any of these proje= cts.


We are working on packaging our work so that it = can be shared with this community.


Thanks

Pankaj

=

On Tue, Oct 25, 2016 at 11:36 AM, Mangi= rish Wagle <vaglomangirish@gmail.com> wrote:
Hi Mark,

Than= ks for your question. So if I understand you correctly, you need kind of lo= ad balancing between identical clusters through a single Mesos master?

With the current setup, from what I understand, we hav= e a separate mesos masters for every cluster on separate clouds. However, i= ts a good investigative topic if we can have single mesos master targeting = multiple identical clusters. We have some work ongoing to use a virtual clu= ster setup with compute resources across clouds to install mesos, but not s= ure if that is what you are looking for.

Regards,<= /div>
Mangirish




On Tue, Oct 25, 2016 at 11:05 AM, Miller, Mark <mmiller@sdsc.edu><= /span> wrote:

Hi all,

=C2=A0

I posed a question to Suresh (see below), and he ask= ed me to put this question on the dev list.

So here it is. I will be grateful for any comments a= bout the issues you all are facing, and what has come up in trying this, as=

It seems likely that this is a much simpler problem = in concept than it is in practice, but its solution has many benefits.

=C2=A0

Here is my question:

A group of us have been discussing how we might simp= lify submitting jobs to different compute resources in our current implemen= tation of CIPRES, and how cloud computing might facilitate this. But none o= f us are cloud experts. As I understand it, the mesos cluster that I have been seeing in the Airavata email thread= s is intended to make it possible to deploy jobs to multiple virtual cluste= rs. I am (we are) wondering if Mesos manages submissions to identical virtu= al clusters on multiple machines, and if that works efficiently.=C2=A0<= /u>

=C2=A0

In our implementation, we have to change the rules t= o run efficiently on different machines, according to gpu availability, and= cores per node. I am wondering how Mesos/ virtual clusters affect those co= nsiderations.

Can mesos create basically identical virtual cluster= s independent of machine?


Thanks for any advice.

=C2=A0

Mark

=C2=A0

=C2=A0

=C2=A0

=C2=A0




--001a114bda32489434053fef9b0e--