Mailing-List: contact mesos-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mesos-dev@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of dchalloner@gmail.com
 designates 209.85.220.182 as permitted sender)
MIME-Version: 1.0
Date: Sat, 20 Apr 2013 12:00:37 -0700
Message-ID: 
 <CA+Enyu51KQkHTPaaxrEw+Ar6PYByxL5v-EHS27ju2Csg56Axag@mail.gmail.com>
Subject: Best way to go about writing a framework for daemon-like tasks
From: David Challoner <dchalloner@gmail.com>
To: mesos-dev@incubator.apache.org
Content-Type: multipart/alternative; boundary=089e01293fe278bcc904dacf7345

--089e01293fe278bcc904dacf7345
Content-Type: text/plain; charset=ISO-8859-1

Hi, new to mesos.   I setup a test cluster in EC2 (which required some
tweaks to the provided scripts - i'll try to send those back in) but I'm
not sure how I should write the framework for what I'd like to achieve.

I'd like to use Mesos to run a dynamically changing list of applications
like so:
[
{ job: app,
  cpu: 3,
  memory: 20g,
  instances: 4},
 { job: app2,
  cpu: 5,
  memory: 30g,
  instances: 2}
]

So I want the framework to pull the list out of a database/redis/zk and run
these apps across the cluster in a round robbin fashion until either the
cluster resources are exhausted or we've satisfied the number of running
instances.   What I'm having trouble groking is how this would look on a
framework/executor level.

With a single framework it seems you're likely to fail using all available
resources given a long running daemon-like tasks:
*framework registered
*node offers 10 cpus
*framework accepts offer and at the time decides to to gives it 8cpu worth
of tasks to run.
*node has 2 cpus left over
*at some point the list of apps changes and a new app allocation is
needed..  Maybe we have a new app that could use those 2 cpus or maybe we
just need to adjust how many of the old apps are running.  If I understand
the docs correctly, the nodes won't re-offer because they've already been
assigned tasks by the framework that will run forever.

Do I maybe submit a new framework for each app type then?  Would that scale
to large numbers of apps?

--089e01293fe278bcc904dacf7345--