mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Chen <...@mesosphere.io>
Subject Re: Batch Scheduler with dependency support
Date Wed, 13 May 2015 20:13:13 GMT
Hi Alex,

Thanks for replying and looks like a really interesting framework! I was
orginally aiming the question back to Aaron as he stated he's looking for a
batch job scheduler.

Do you have some rough stats how many jobs or data you guys are computing
with Stolos?

Tim



On Wed, May 13, 2015 at 1:04 PM, Alex Gaudio <adgaudio@gmail.com> wrote:

> Hi Tim (and everyone else!),
>
> I am the primary author of Stolos.  We use Stolos to run all of our batch
> jobs on Mesos.  The batch jobs are scripts we can run from the
> command-line.  Scripts range from bash scripts, Spark jobs and R scripts.
>
> It's a great tool for us because, unlike Chronos, it lets us define a
> script as stage in a dependency chain, where the script can run with
> different parameters for different dependency contexts.  (The closest usage
> of this would be to have many Chronos servers, though this does not work in
> all cases).
>
> The tool is a critical component of Sailthru's data science
> infrastructure, but I believe we are the only people who use the tool right
> now.
>
> If you are interested in learning more, I'm happy to invest time to talk
> more about Stolos, what it does and how we use it!
>
> Alex
>
> On Wed, May 13, 2015 at 2:02 PM Tim Chen <tim@mesosphere.io> wrote:
>
>> How are you running your batch jobs? Is the batch job script/executable
>> an in-house app?
>>
>> Tim
>>
>> On Wed, May 13, 2015 at 9:46 AM, Andras Kerekes <
>> andras.kerekes@ishisystems.com> wrote:
>>
>>> You might want to have a look at stolos too:
>>>
>>>
>>>
>>> https://github.com/sailthru/stolos
>>>
>>>
>>>
>>> Andras
>>>
>>>
>>>
>>>
>>>
>>> *From:* Aaron Carey [mailto:acarey@ilm.com]
>>> *Sent:* Wednesday, May 13, 2015 11:54 AM
>>> *To:* user@mesos.apache.org
>>> *Subject:* RE: Batch Scheduler with dependency support
>>>
>>>
>>>
>>> Thanks! I hadn't come across that one before :)
>>> ------------------------------
>>>
>>> *From:* jeffschroed@gmail.com [jeffschroed@gmail.com] on behalf of Jeff
>>> Schroeder [jeffschroeder@computer.org]
>>> *Sent:* 13 May 2015 16:39
>>> *To:* user@mesos.apache.org
>>> *Subject:* Re: Batch Scheduler with dependency support
>>>
>>> Lookup Hubspot's Singularity
>>>
>>> On Wednesday, May 13, 2015, Aaron Carey <acarey@ilm.com> wrote:
>>>
>>> Thanks Jeff,
>>>
>>> Any other options around as well?
>>> ------------------------------
>>>
>>> *From:* jeffschroed@gmail.com <http://UrlBlockedError.aspx> [
>>> jeffschroed@gmail.com <http://UrlBlockedError.aspx>] on behalf of Jeff
>>> Schroeder [jeffschroeder@computer.org <http://UrlBlockedError.aspx>]
>>> *Sent:* 13 May 2015 14:12
>>> *To:* user@mesos.apache.org <http://UrlBlockedError.aspx>
>>> *Subject:* Batch Scheduler with dependency support
>>>
>>> It does both just as well, along with cron-like functionality. It is
>>> harder to install and takes a bit more understanding however. The official
>>> tutorial is a process that loops 100 times and then exits.
>>>
>>>
>>>
>>> http://aurora.apache.org/documentation/latest/tutorial/#the-script
>>>
>>> Aurora is pretty much a superset of most other generic frameworks sans
>>> maybe hubspot's singularity.
>>>
>>>
>>> On Wednesday, May 13, 2015, Aaron Carey <acarey@ilm.com
>>> <http://UrlBlockedError.aspx>> wrote:
>>>
>>> I was under the impression Aurora was for long running services? Is it
>>> suitable for scheduling one of batch processes too?
>>>
>>> thanks,
>>> Aaron
>>> ------------------------------
>>>
>>> *From:* jeffschroed@gmail.com [jeffschroed@gmail.com] on behalf of Jeff
>>> Schroeder [jeffschroeder@computer.org]
>>> *Sent:* 13 May 2015 13:12
>>> *To:* user@mesos.apache.org
>>> *Subject:* Re: Batch Scheduler with dependency support
>>>
>>> Apache Aurora does this and you can be explicit about the ordering
>>>
>>> On Wednesday, May 13, 2015, Aaron Carey <acarey@ilm.com> wrote:
>>>
>>> Hi All,
>>>
>>> I was just wondering if anyone out there knew of a good mesos batch
>>> scheduler which supports dependencies between tasks? (ie Task B cannot run
>>> until Task A is complete)
>>>
>>> Thanks,
>>> Aaron
>>>
>>>
>>>
>>> --
>>> Text by Jeff, typos by iPhone
>>>
>>>
>>>
>>> --
>>> Text by Jeff, typos by iPhone
>>>
>>>
>>>
>>> --
>>> Text by Jeff, typos by iPhone
>>>
>>
>>

Mime
View raw message