airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shenoy, Gourav Ganesh" <goshe...@indiana.edu>
Subject Re: Mesos based meta-scheduling for Airavata
Date Sat, 08 Oct 2016 02:56:46 GMT
Hi dev,

I have been exploring different frameworks for Mesos which would help our use-case of providing
Airavata the capability to run jobs in a Mesos based ecosystem. In particular, I have been
playing around with Marathon & Chronos and I am now going to be working on Apache Aurora.

I have summarized my understanding about Mesos, Marathon & Chronos below. I will send
out a separate email about Aurora later.

Apache Mesos:


·         Apache Mesos is an open-source cluster manager, in the sense that it helps deploy
& manage different frameworks (or applications) in a large clustered environment easily.

·         Mesos provides the ability to utilize underlying shared pool of nodes as a single
compute unit – That is, it can run many applications on these nodes efficiently.

·         Mesos uses the concept of “offers” for scheduling and running jobs on the underlying
nodes. When a framework (application) wants to run computations/jobs on the cluster, Mesos
will decide how many resources it will “offer” that framework based on the availability.
The framework will then decide which resources to use from the offer, and subsequently run
the computation/job on that resource.

·         In a typical cluster, you will have 3 or more Mesos masters & multiple Mesos
slaves. Multiple mesos masters help in providing high availability – if one master goes
down, Mesos will reelect a new leader (master) – using Zookeeper.

·         The task mentioned above of providing “offers” to frameworks is done by a master,
whereas the slaves are the ones who run these computations.


·         Some additional points:

o    I built a Mesos cluster with 3 masters & 2 slaves on EC2.

o    Each master & slave have 1GB of RAM & 1vCPU with 20GB of disk space.

Marathon:


·         Marathon is considered a framework that runs on top of Mesos. It is a container
orchestration platform for Mesos and essentially acts as a service scheduler.

·         It is named “marathon” because it is intended for long running applications.
That is, Marathon makes sure that the service it is running never stops – if a service goes
down or the slave on which the service is run dies, marathon keeps re-starting it on different
slaves.

·         In some sense Marathon is very good for ensuring high availability of services.
That is, instead of running services directly on Mesos, run it in Marathon if you never want
it to die.
Note: You can decide to run a service on multiple slave nodes and if resources on these slaves
are available, Mesos will “offer” them to Marathon.

·         It is called a container orchestration platform because it “launches” these
services inside a container – either Docker OR Mesos container.

·         In my opinion it is not a suitable “job scheduler” for Airavata because in
Airavata we need to run a job and get the output rather than keeping it running always. Instead,
we can run other schedulers – chronos/aurora as a service in Marathon.

Chronos:


·         Chronos is a Cron scheduler for Mesos. It is good for running scheduled jobs –
jobs that need to be run for a certain number of times, repeatedly after certain intervals.

·         Chronos also provides the ability to add dependencies between jobs – That is,
if a job1 is dependent on another job2 then it will run job1 first and then run job2 after
job1 completes. It also builds a Directed Acyclic Graph (DAG) based on these dependencies.

·         Similar to Marathon, Chronos receives “offers” from Mesos master whenever it
needs to run a job on Mesos.

·         Again, I found that Chronos does not fit the Airavata use-case since I could not
find a way to run one-off jobs via Chronos – you need to specify interval time for Chronos,
& Chronos then re-runs the job after that interval is complete (even if you decide to
specify num. of repetitions=1).


Some additional points:

·         Marathon & Chronos both have REST API support – eg: you can submit jobs via
APIs along with other interactions such as list jobs, etc.

·         I installed Marathon & Chronos frameworks on the Mesos master nodes. This is
how their health looks like on the Mesos dashboard:

[cid:image002.png@01D220EE.1346FB90]
                As you can see, there are 3 active tasks running in Chronos & 4 active
tasks (long running) in Marathon.


·         I also installed Chronos as a service inside Marathon, and this is how it looks
like in the Marathon UI:


[cid:image004.png@01D220EE.1346FB90]
Interestingly, Chronos (as a service in Marathon) was smart enough to identify the jobs submitted
via Chronos (as a framework on Mesos) & vice-versa.


·         Also, Mesos dashboard lists the active tasks it is running & details about
which slave the task is running on. It also lists Completed tasks. The “Sandbox” gives
you access to the stdout/stderr files for the tasks as well as any other directories that
were created as part of the task.

[cid:image005.png@01D220EE.1346FB90]

Pardon me for this long email. Next, I will explore Apache Aurora which seems a better fit
for Airavata use-case because it provides the features that Chronos supports, as well as can
run one-off jobs.

Thanks and Regards,
Gourav Shenoy

From: "Shenoy, Gourav Ganesh" <goshenoy@indiana.edu>
Reply-To: "dev@airavata.apache.org" <dev@airavata.apache.org>
Date: Friday, September 23, 2016 at 4:43 PM
To: "dev@airavata.apache.org" <dev@airavata.apache.org>
Subject: Mesos based meta-scheduling for Airavata

Hi Dev,

I am working on this project of building a Mesos based meta-scheduler for Airavata, along
with Shameera & Mangirish. Here is the jira link: https://issues.apache.org/jira/browse/AIRAVATA-2082.


·         We have identified some tasks that would be needed for achieving this, and at the
higher level it would consist of:

1.      Resource provisioning – We need to provision resources on cloud & hpc infrastructures
such as EC2, Jetstream, Comet, etc.

2.      Building a cluster – Deploying a Mesos cluster on set of nodes obtained from (1)
above for task management.

3.      Selecting a scheduler – We need to investigate the scheduler to use with Mesos cluster.
Some of the options are Marathon, Aurora. But we need to find one that suits our needs of
running serial as well as parallel (MPI) jobs.

4.      Installing & running applications on this cluster – Once the cluster has been
deployed and a scheduler choice made, we need to be able to install and run applications on
this cluster using Airavata.


·         Until now we were able to look into the following:

o   Resource provisioning:

§  We explored several options of provisioning resources – using cloud libraries as well
as via ansible scripts.

§  We built a OpenStack4J Java module which would provision instances on OpenStack based
clouds (eg: Jetstream).

§  We also built a CloudBridge Python module for provisioning EC2 instances on Amazon. CloudBridge
can also be used to provision instances on OpenStack

§  We wrote Ansible scripts for bringing up instances on both AWS and OpenStack based clouds.


§  Key Points: CloudBridge, OpenStack4J are powerful libraries for resource provisioning,
but currently they do single-instance provisioning, and not support templated boot options
such as CloudFormation (for AWS) & Heat (for OpenStack).


o   Building a cluster:

§  We wrote Ansible script for deploying a Mesos-Marathon cluster on a set of nodes. This
script will install necessary dependencies such as Zookeeper.

§  We tested this on OpenStack based clouds & on EC2.

§  OpenStack Magnum provides excellent support for doing resource provisioning & deploying
mesos cluster, but we are running into some problems while trying it.


o   Installing a scheduler:

§  Our Ansible script is currently installing Marathon as the scheduler on Mesos. We haven’t
yet submitted jobs using Marathon.


·         Although not finalized, but we are inclined towards using Ansible approach for
the above, as Ansible also provides Python APIs and which will allow us to integrate it with
Airavata via Thrift. Hence we will be able to easily invoke the Ansible scripts from code
without needing to use the command-line interface.


·         We are also progressively working on some work-items such as:

o   Exploring options to provision and deploy a Mesos-Marathon cluster on HPC systems such
as Comet. The challenge would be to use Ansible to provision resources and deploy the cluster.
Once we have a cluster, we can try running applications.

o   Exploring different scheduler options for running serial and parallel (MPI) jobs on such
heterogeneous clusters.

o   Exploring orchestration options such as OpenStack Heat, AWS CloudFormation, OpenStack
Magnum, etc.

Any suggestions and comments are highly appreciated.

Thanks and Regards,
Gourav Shenoy



Mime
View raw message