mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Gaudio <adgau...@gmail.com>
Subject Re: Batch Scheduler with dependency support
Date Wed, 13 May 2015 20:04:20 GMT
Hi Tim (and everyone else!),

I am the primary author of Stolos.  We use Stolos to run all of our batch
jobs on Mesos.  The batch jobs are scripts we can run from the
command-line.  Scripts range from bash scripts, Spark jobs and R scripts.

It's a great tool for us because, unlike Chronos, it lets us define a
script as stage in a dependency chain, where the script can run with
different parameters for different dependency contexts.  (The closest usage
of this would be to have many Chronos servers, though this does not work in
all cases).

The tool is a critical component of Sailthru's data science infrastructure,
but I believe we are the only people who use the tool right now.

If you are interested in learning more, I'm happy to invest time to talk
more about Stolos, what it does and how we use it!

Alex

On Wed, May 13, 2015 at 2:02 PM Tim Chen <tim@mesosphere.io> wrote:

> How are you running your batch jobs? Is the batch job script/executable an
> in-house app?
>
> Tim
>
> On Wed, May 13, 2015 at 9:46 AM, Andras Kerekes <
> andras.kerekes@ishisystems.com> wrote:
>
>> You might want to have a look at stolos too:
>>
>>
>>
>> https://github.com/sailthru/stolos
>>
>>
>>
>> Andras
>>
>>
>>
>>
>>
>> *From:* Aaron Carey [mailto:acarey@ilm.com]
>> *Sent:* Wednesday, May 13, 2015 11:54 AM
>> *To:* user@mesos.apache.org
>> *Subject:* RE: Batch Scheduler with dependency support
>>
>>
>>
>> Thanks! I hadn't come across that one before :)
>> ------------------------------
>>
>> *From:* jeffschroed@gmail.com [jeffschroed@gmail.com] on behalf of Jeff
>> Schroeder [jeffschroeder@computer.org]
>> *Sent:* 13 May 2015 16:39
>> *To:* user@mesos.apache.org
>> *Subject:* Re: Batch Scheduler with dependency support
>>
>> Lookup Hubspot's Singularity
>>
>> On Wednesday, May 13, 2015, Aaron Carey <acarey@ilm.com> wrote:
>>
>> Thanks Jeff,
>>
>> Any other options around as well?
>> ------------------------------
>>
>> *From:* jeffschroed@gmail.com <http://UrlBlockedError.aspx> [
>> jeffschroed@gmail.com <http://UrlBlockedError.aspx>] on behalf of Jeff
>> Schroeder [jeffschroeder@computer.org <http://UrlBlockedError.aspx>]
>> *Sent:* 13 May 2015 14:12
>> *To:* user@mesos.apache.org <http://UrlBlockedError.aspx>
>> *Subject:* Batch Scheduler with dependency support
>>
>> It does both just as well, along with cron-like functionality. It is
>> harder to install and takes a bit more understanding however. The official
>> tutorial is a process that loops 100 times and then exits.
>>
>>
>>
>> http://aurora.apache.org/documentation/latest/tutorial/#the-script
>>
>> Aurora is pretty much a superset of most other generic frameworks sans
>> maybe hubspot's singularity.
>>
>>
>> On Wednesday, May 13, 2015, Aaron Carey <acarey@ilm.com
>> <http://UrlBlockedError.aspx>> wrote:
>>
>> I was under the impression Aurora was for long running services? Is it
>> suitable for scheduling one of batch processes too?
>>
>> thanks,
>> Aaron
>> ------------------------------
>>
>> *From:* jeffschroed@gmail.com [jeffschroed@gmail.com] on behalf of Jeff
>> Schroeder [jeffschroeder@computer.org]
>> *Sent:* 13 May 2015 13:12
>> *To:* user@mesos.apache.org
>> *Subject:* Re: Batch Scheduler with dependency support
>>
>> Apache Aurora does this and you can be explicit about the ordering
>>
>> On Wednesday, May 13, 2015, Aaron Carey <acarey@ilm.com> wrote:
>>
>> Hi All,
>>
>> I was just wondering if anyone out there knew of a good mesos batch
>> scheduler which supports dependencies between tasks? (ie Task B cannot run
>> until Task A is complete)
>>
>> Thanks,
>> Aaron
>>
>>
>>
>> --
>> Text by Jeff, typos by iPhone
>>
>>
>>
>> --
>> Text by Jeff, typos by iPhone
>>
>>
>>
>> --
>> Text by Jeff, typos by iPhone
>>
>
>

Mime
View raw message