mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Olivier Sallou <olivier.sal...@irisa.fr>
Subject Re: GenOuest makes use of Mesos
Date Thu, 10 Dec 2015 11:08:18 GMT

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256



On 12/09/2015 03:41 PM, Joao Ribeiro wrote:
> Hi Oliver,
>
> GoDocker looks very promising. I have been looking for a a framework
to substitute AWS Lambda for batch jobs scheduling based on docker for
flexibility. That said, I would like to know your thought on this to
better understand if GoDocker could do the job.
> A couple of things I would need to know better:
>
> - How fast can GoDocker launch a job? Can GoDocker pre populate all
nodes with the container and files needed for the job to achieve <500ms
launches? I realize this depends a lot on the Job size, just looking for
a generic view before getting into full testing.

Fast... depends on number of pending jobs. There is at first a scheduler
based on user/project previous usage, priorities and quota checks to
reorder the jobs before assigning them to nodes. The more you have
pending jobs, the more it will take for this step. If you select a FIFO
scheduler instead of Fair share scheduler it is of course quite more
faster. Then it depends also on Mesos offers (we wait for a mesos offer
to assign jobs to nodes).

We cannot prepopulate containers to exec scripts in containers
afterwards. We always start a new container which has a dedicated
directory to write some output files (besides optional home directory
etc...). Once container is started on node, we cannot access anymore the
container to "submit" tasks in the container, though I think this could
be managed with a special Executor in Mesos instead of using the default
Executor.

The logic is:

get pending jobs
reorder according to prio/quota etc...
get mesos offer and assign jobs to mesos nodes
one (or many) watcher checks job status
    if job over, update job status and meta-data


one possiblity of course would be to execute in the container a program
that fetches job commands or files to use from a db/queue available
elsewhere, and exit when queue is empty. This speeds up the process of
many files, but you won't get details "per job" .

> - Is it possible to extend GoDocker to enable usage metrics reports? This could be achieved
by
simply logging a few metrics relating to the user and job.
For job metrics, GoDocker makes use of cAdvisor for live monitoring.
cAdvisor also has a prometheus endpoint (that we use in our premises) to
get live and past monitoring of containers.
GoDocker provides itself a prometheus endpoint for global statistics. We
also store in job meta data its start/end time, requested cpu/ram, and
real job duration.


Olivier
>
>
> { name: “*João Ribeiro*”,
>   email: “jonnybgod@gmail.com <mailto:jonnybgod@gmail.com>”,
>   phone: “+351 916 002 675 <tel:+351916002675>”,
>   skype: “jmn_ribeiro <skype:jmn_ribeiro?add>”,
>   linkedin: “https://pt.linkedin.com/in/jmnribeiro”
>   github: “https://github.com/JonnyBGod” }
>
>> On 09 Dec 2015, at 10:22, Elouan Keryell-Even
<elouan.keryell@gmail.com <mailto:elouan.keryell@gmail.com>> wrote:
>>
>> Thanks for the insight, very interesting indeed.
>>
>> Elouan Keryell-Even
>>
>> Software Engineer @ Atos Integration
>>
>> Toulouse, France
>>
>>
>> 2015-12-09 11:08 GMT+01:00 Olivier Sallou <olivier.sallou@irisa.fr
<mailto:olivier.sallou@irisa.fr>>:
>>
>>
>>
>>     -------------------------
>>
>>         *De: *"Elouan Keryell-Even" <elouan.keryell@gmail.com
<mailto:elouan.keryell@gmail.com>>
>>         *À: *user@mesos.apache.org <mailto:user@mesos.apache.org>
>>         *Envoyé: *Mercredi 9 Décembre 2015 10:48:10
>>         *Objet: *Re: GenOuest makes use of Mesos
>>
>>         Hi Olivier,
>>
>>         I've just read the presentation on your project's webpage,
and it seems cool! I'm curious about the following mentioned feature:
"Optional mount of user home or other shared directories in container".
Does your framework take care of remotely copying the home directory
onto the node where the container is going to run, or does the directory
have to be already available (as a shared directory for example).
>>
>>     We focus to "replace" classical computing cluster frameworks like
Sun Grid Engine. So home directories etc... are shared among nodes (nfs,
etc...), like in a usual cluster environment.
>>     The mount of "user home dir" is based on an Auth/ACL
plugin/configuration. Admin specifies the available mounts for all
users, or on a project basis. Then the Auth/ACL plugin maps those
volumes to real available directories (if using the LDAP auth plugin,
then get homeDirectory from LDAP and add as a volume to container).
>>
>>         Otherwise, we are currently using framework Chronos for our
Batch oriented containers, and your framework seems to fit the same
spot. If you have some experience with Chronos, I'd be interested in a
brief comparison of both frameworks. From what I understand, GODocker
offers scheduling policy's customization, which I don't think Chronos does.
>>
>>     There is indeed scheduling algorithms plugin mechanism. An
interesting one is the fair-share policy imlpementation, where tasks are
scheduler (ordered) based on previous user/project consumption.
>>
>>     Difference with Chronos, is Chronos is made for cron like jobs.
GODocker is a batch submission tool. User specifies the computing script
he wants to execute and it is executed "immediatly". The scheduling is a
matter of putting priorities when executing jobs (if users submits 1000
jobs but I have only 100 slots available). We focus on users submitting
"many" jobs or many users submitting jobs, each job could be a computing
task of a few seconds or several days.
>>
>>
>>
>>     Olivier
>>
>>
>>         Thanks for your attention,
>>
>>         Elouan Keryell-Even
>>
>>         Software Engineer @ Atos Integration
>>
>>         Toulouse, France
>>
>>
>>         2015-12-08 14:49 GMT+01:00 Olivier Sallou
<olivier.sallou@irisa.fr <mailto:olivier.sallou@irisa.fr>>:
>>
>>             Hi,
>>             the GenOuest (http://www.genouest.org
<http://www.genouest.org/>) academic lab is now using Mesos
>>             in production in its core facility to manage scientists
computing tasks
>>             (for bioinformatics)
>>             To do so, we have developed a new mesos framework, GoDocker
>>             (http://www.genouest.org/godocker) to submit batch
computing scripts on
>>             premises. It mounts users home directory or other shared
resources to
>>             execute jobs in Docker containers, using Mesos as main
scheduler.
>>             GoDocker schedules the jobs according to user/groups
priorities and
>>             quotas and provides a CLI, a REST web interface and a
partial DRMAA
>>             library support. Framewok is open source.
>>
>>             Thanks for adding us to the Mesos fellows ;-)
>>
>>             Regards
>>
>>             Olivier (GenOuest core developer member)
>>
>>             --
>>             Olivier Sallou
>>             IRISA / University of Rennes 1
>>             Campus de Beaulieu, 35000 RENNES - FRANCE
>>             Tel: 02.99.84.71.95 <tel:02.99.84.71.95>
>>
>>             gpg key id: 4096R/326D8438  (keyring.debian.org
<http://keyring.debian.org/>)
>>             Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC
68DB 326D 8438
>>
>>
>>
>>
>>
>

- -- 
Olivier Sallou
IRISA / University of Rennes 1
Campus de Beaulieu, 35000 RENNES - FRANCE
Tel: 02.99.84.71.95

gpg key id: 4096R/326D8438  (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iQIcBAEBCAAGBQJWaV0hAAoJEHjcaNsybYQ4WMIP+wbL9AbiYJzNKbLVT7NJvHPn
f2/gZoxefjZxt1qLhBGtu9LglhptAyliZ3lYe7g8LGdUNzsHcmBXSDwk2lMNjYdj
IT/FbWLXU7KeKcUixM2SG7uw3nrtrkwBOpYa2lPEDwPsSKZlbTsyaKsSKDJlO7gb
EQBXKyRVtiGZOnt/WbgCNT1+EfwzN07DfYd9wQnVXt2o5ouIuYc9JRNMYBYYjRnP
rBFjkE4YoquPC0W7MV62I7C/C4HsEiorlvm1tbPGv0k20Ep7+Nv6LeY/05a+n0Gv
mImBNvzjELt3KNYhaZb22o5aGDdSSKeOYt3hs83T7AoSkLThHNWwDdRmB4d/BwxP
ieFWTtprTl+gPf3zV4ERvlGQKap+agNY9BIgZnrEV9XBm+qR/tgo0jQ8cV9zKSJT
Di7mmDSDozdEci7TFbpIl6WrUjet84Cfjrfr7rM7wmgXWmYNLNZoeXNS03w426Cz
Zwy4b8CJuB5g1IIsaqOMLoUVB73AFtRH9OTEnz07dgio3C/PwwkZPaMt8nunZKoV
jA7cCOufQjBLpX9kwscasYplAmwwkFjrR6chEPu4rFBPvuR31ce8gTmkbvo45m9N
nsIZ9TNZB+aXFNiSa9GSnCtNAfMxmh02bvGbSJ2G3ILKUQqLeQBpXk0OL5Cz9G4O
z3nxqyzo438A2mDK4ihK
=m7/b
-----END PGP SIGNATURE-----


Mime
View raw message