Return-Path: X-Original-To: apmail-mesos-user-archive@www.apache.org Delivered-To: apmail-mesos-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BF85B10AE7 for ; Fri, 27 Sep 2013 19:41:14 +0000 (UTC) Received: (qmail 68523 invoked by uid 500); 27 Sep 2013 19:41:13 -0000 Delivered-To: apmail-mesos-user-archive@mesos.apache.org Received: (qmail 68508 invoked by uid 500); 27 Sep 2013 19:41:12 -0000 Mailing-List: contact user-help@mesos.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@mesos.apache.org Delivered-To: mailing list user@mesos.apache.org Received: (qmail 68500 invoked by uid 99); 27 Sep 2013 19:41:10 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Sep 2013 19:41:10 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of tahasam@gmail.com designates 209.85.220.50 as permitted sender) Received: from [209.85.220.50] (HELO mail-pa0-f50.google.com) (209.85.220.50) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Sep 2013 19:41:07 +0000 Received: by mail-pa0-f50.google.com with SMTP id fb1so3167333pad.23 for ; Fri, 27 Sep 2013 12:40:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=fL9+I0q9FyKurPeCUlQwQisItpLNUSuV4nKX1uWDEfI=; b=Pb2T475Hc6hlR+piQG2cx3WZGqXXFiAHhHI/51lnivJyrMzJ03+H7QuC8fJqMLIbDq exZhJQ6ZqN8VuEm00OWZnMLmn3vGmDKZ0FUp8KHvEaHhtn6xqznnohQUh0iQISinI7yn wWa6hWK+t6t0viZQapU/r2LzZBztYdAn18Rw3ymk6lqVzzpJ+rDcU4t71aWr/Bf2soQU xE93dst+baZ4/r9jgCPMCi/y6HnGguZlnCkOVpsukJ3YSLXXcW+iXP9srL8ntXuSkU0c pGSo8sKBMFehMpvxgLAkIaGzHhDnP5HYOJ8ES5KOnTGV2dAli9E6KnmQO3sre2JLj+3T OBOA== X-Received: by 10.68.135.132 with SMTP id ps4mr8980966pbb.171.1380310846745; Fri, 27 Sep 2013 12:40:46 -0700 (PDT) MIME-Version: 1.0 Received: by 10.70.132.167 with HTTP; Fri, 27 Sep 2013 12:40:26 -0700 (PDT) In-Reply-To: References: From: Sam Taha Date: Fri, 27 Sep 2013 15:40:26 -0400 Message-ID: Subject: Re: Aurora, Marathon and long lived job frameworks To: user@mesos.apache.org Content-Type: multipart/alternative; boundary=e89a8ffbad61b333eb04e762a94f X-Virus-Checked: Checked by ClamAV on apache.org --e89a8ffbad61b333eb04e762a94f Content-Type: text/plain; charset=ISO-8859-1 While still in active development, I expect JobServer to match some of the criteria you describe once Mesos integration is complete. It currently supports these features for static node clusters. With mesos integration, it will have dynamic clustering capability while still retaining the enterprise type job scheduling/monitoring/tracking...etc features. Thanks, Sam Taha http://www.grandlogic.com On Fri, Sep 27, 2013 at 12:59 PM, Dan Colish wrote: > > On Fri, Sep 27, 2013 at 9:04 AM, Damien Hardy wrote: > >> Hello, >> >> What about chronos http://airbnb.github.io/chronos/ >> >> > Yes, I evaluated chronos and it was not clear to me how it matches my > selection criteria. It might be my unfamiliarity with the framework rather > than a lack of features. Is there anyone who could elaborate more? > > >> Best regards, >> >> >> 2013/9/27 Dan Colish >> >>> I have been working on an internal project for executing a large number >>> of jobs across a cluster for the past couple of months and I am currently >>> doing a spike on using mesos for some of the cluster management tasks. The >>> clear prior art winners are Aurora and Marathon, but in both cases they >>> fall short of what I need. >>> >>> In aurora's case, the software is clearly very early in the open >>> sourcing process and as a result it missing significant pieces. The biggest >>> missing piece is the actual execution framework, Thermos. [That is what I >>> assume thermos does. I have no internal knowledge to verify that >>> assumption] Additionally, Aurora is heavily optimized for a high user count >>> and large number of incoming jobs. My use case is much simpler. There is >>> only one effective user and we have a small known set of jobs which need to >>> run. >>> >>> On the other hand, Marathon is not designed for job execution if job is >>> defined to be a smaller unit of work. Instead, Marathon self-describes as a >>> meta-framework for deploying frameworks to a mesos cluster. A job to >>> marathon is the framework that runs. I do not think Marathon would be a >>> good fit for managing the my task execution and retry logic. It is designed >>> to run at on as a sub-layer of the cluster's resource allocation scheduler >>> and its abstractions follow suit. >>> >>> For my needs Aurora does appear to be a much closer fit than Marathon, >>> but neither is ideal. Since that is the case, I find myself left with a >>> rough choice. I am not thrilled with the prospect of yet another framework >>> for Mesos, but there is a lot of work which I have already completed for my >>> internal project that would need to reworked to fit with Aurora. Currently >>> my project can support the following features. >>> >>> * Distributed job locking - jobs cannot overlap >>> * Job execution delay queue - jobs can be run immediately or after a >>> delay >>> * Job preemption >>> * Job success/failure tracking >>> * Garbage collection of dead jobs >>> * Job execution failover - job is retried on a new executor >>> * Executor warming - min # of executors idle >>> * Executor limits - max # of executors available >>> >>> My plan for integration with mesos is to adapt the job manager into a >>> mesos scheduler and my execution slaves into a mesos executor. At that >>> point, my framework will be able to run on the mesos cluster, but I have a >>> few concerns about how to allocated and release resources that the >>> executors will use over the lifetime of the cluster. I am not sure whether >>> it is better to be greedy early on in the frameworks life-cycle or to >>> decline resources initially and scale the framework's slaves when jobs >>> start coming in. Additionally, the relationship between the executor and >>> its associated driver are not immediately clear to me. If I am reading the >>> code correctly, they do not provide a way to stop a task in progress short >>> of killing the executor process. >>> >>> I think that mesos will be a nice feature to add to my project and I >>> would really appreciate any feedback from the community. I will provide >>> progress updates as I continue work on my experiments. >>> >> >> >> >> -- >> Damien HARDY >> > > --e89a8ffbad61b333eb04e762a94f Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
While still in active development, I expect JobServer to m= atch some of the criteria you describe once Mesos integration is complete. = It currently supports these features for static node clusters. With mesos i= ntegration, it will have dynamic clustering capability while still retainin= g the enterprise type job scheduling/monitoring/tracking...etc features.

Thanks,
Sam Taha


<= br>
On Fri, Sep 27, 2013 at 12:59 PM, Dan Colish = <dcolish@urbanairship.com> wrote:

On Fri, Sep 27, 2013 at 9:= 04 AM, Damien Hardy <dhardy@viadeoteam.com> wrote:
Hello,

What about chronos http://airbnb.github.io/chronos/


Yes, I evaluated chr= onos and it was not clear to me how it matches my selection criteria. It mi= ght be my unfamiliarity with the framework rather than a lack of features. = Is there anyone who could elaborate more?=A0
=A0
Be= st regards,


2013/9/27 Dan Colish <dcolish@urbanairship.com>
I have been working on an internal project for executing a= large number of jobs across a cluster for the past couple of months and I = am currently doing a spike on using mesos for some of the cluster managemen= t tasks. The clear prior art winners are Aurora and Marathon, but in both c= ases they fall short of what I need.=A0

In aurora's case, the software is clearly very early in = the open sourcing process and as a result it missing significant pieces. Th= e biggest missing piece is the actual execution framework, Thermos. [That i= s what I assume thermos does. I have no internal knowledge to verify that a= ssumption] Additionally, Aurora is heavily optimized for a high user count = and large number of incoming jobs. My use case is much simpler. There is on= ly one effective user and we have a small known set of jobs which need to r= un.=A0

On the other hand, Marathon is not designed for job exe= cution if job is defined to be a smaller unit of work. Instead, Marathon se= lf-describes as a meta-framework for deploying frameworks to a mesos cluste= r. A job to marathon is the framework that runs. I do not think Marathon wo= uld be a good fit for managing the my task execution and retry logic. It is= designed to run at on as a sub-layer of the cluster's resource allocat= ion scheduler and its abstractions follow suit.

For my needs Aurora does appear to be a much closer fit= than Marathon, but neither is ideal. Since that is the case, I find myself= left with a rough choice. I am not thrilled with the prospect of yet anoth= er framework for Mesos, but there is a lot of work which I have already com= pleted for my internal project that would need to reworked to fit with Auro= ra. Currently my project can support the following features.

* Distributed job locking - jobs cannot overlap=A0
* Job execution delay queue - jobs can be run immediately or after a = delay
* Job preemption
* Job success/failure tracking
* Garbage collection of dead jobs
* Job execution failover -= job is retried on a new executor
* Executor warming - min # of executors idle
* Executor limi= ts - max # of executors available

My plan for inte= gration with mesos is to adapt the job manager into a mesos scheduler and m= y execution slaves into a mesos executor. At that point, my framework will = be able to run on the mesos cluster, but I have a few concerns about how to= allocated and release resources that the executors will use over the lifet= ime of the cluster. I am not sure whether it is better to be greedy early o= n in the frameworks life-cycle or to decline resources initially and scale = the framework's slaves when jobs start coming in. Additionally, the rel= ationship between the executor and its associated driver are not immediatel= y clear to me. If I am reading the code correctly, they do not provide a wa= y to stop a task in progress short of killing the executor process.=A0

I think that mesos will be a nice feature to add to my = project and I would really appreciate any feedback from the community. I wi= ll provide progress updates as I continue work on my experiments.



--
Damien HARDY


--e89a8ffbad61b333eb04e762a94f--