mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim St Clair <tstcl...@redhat.com>
Subject Re: Batch Scheduler with dependency support
Date Wed, 13 May 2015 20:12:59 GMT
Hi Alex, 

Have you by chance integrated with any of the tradition batch DAG systems? 

http://pegasus.isi.edu/ , http://ccl.cse.nd.edu/software/makeflow/ 

I keep longing for folks with decades of experience in HTC&HPC to chime in "on-list".


Subtle nudge ;-) 
Tim 

----- Original Message -----

> From: "Alex Gaudio" <adgaudio@gmail.com>
> To: user@mesos.apache.org
> Sent: Wednesday, May 13, 2015 3:04:20 PM
> Subject: Re: Batch Scheduler with dependency support

> Hi Tim (and everyone else!),

> I am the primary author of Stolos. We use Stolos to run all of our batch jobs
> on Mesos. The batch jobs are scripts we can run from the command-line.
> Scripts range from bash scripts, Spark jobs and R scripts.

> It's a great tool for us because, unlike Chronos, it lets us define a script
> as stage in a dependency chain, where the script can run with different
> parameters for different dependency contexts. (The closest usage of this
> would be to have many Chronos servers, though this does not work in all
> cases).

> The tool is a critical component of Sailthru's data science infrastructure,
> but I believe we are the only people who use the tool right now.

> If you are interested in learning more, I'm happy to invest time to talk more
> about Stolos, what it does and how we use it!

> Alex

> On Wed, May 13, 2015 at 2:02 PM Tim Chen < tim@mesosphere.io > wrote:

> > How are you running your batch jobs? Is the batch job script/executable an
> > in-house app?
> 

> > Tim
> 

> > On Wed, May 13, 2015 at 9:46 AM, Andras Kerekes <
> > andras.kerekes@ishisystems.com > wrote:
> 

> > > You might want to have a look at stolos too:
> > 
> 

> > > https://github.com/sailthru/stolos
> > 
> 

> > > Andras
> > 
> 

> > > From: Aaron Carey [mailto: acarey@ilm.com ]
> > 
> 
> > > Sent: Wednesday, May 13, 2015 11:54 AM
> > 
> 
> > > To: user@mesos.apache.org
> > 
> 
> > > Subject: RE: Batch Scheduler with dependency support
> > 
> 

> > > Thanks! I hadn't come across that one before :)
> > 
> 

> > > From: jeffschroed@gmail.com [ jeffschroed@gmail.com ] on behalf of Jeff
> > > Schroeder [ jeffschroeder@computer.org ]
> > 
> 
> > > Sent: 13 May 2015 16:39
> > 
> 
> > > To: user@mesos.apache.org
> > 
> 
> > > Subject: Re: Batch Scheduler with dependency support
> > 
> 

> > > Lookup Hubspot's Singularity
> > 
> 

> > > On Wednesday, May 13, 2015, Aaron Carey < acarey@ilm.com > wrote:
> > 
> 

> > > Thanks Jeff,
> > 
> 

> > > Any other options around as well?
> > 
> 

> > > From: jeffschroed@gmail.com [ jeffschroed@gmail.com ] on behalf of Jeff
> > > Schroeder [ jeffschroeder@computer.org ]
> > 
> 
> > > Sent: 13 May 2015 14:12
> > 
> 
> > > To: user@mesos.apache.org
> > 
> 
> > > Subject: Batch Scheduler with dependency support
> > 
> 

> > > It does both just as well, along with cron-like functionality. It is
> > > harder
> > > to install and takes a bit more understanding however. The official
> > > tutorial
> > > is a process that loops 100 times and then exits.
> > 
> 

> > > http://aurora.apache.org/documentation/latest/tutorial/#the-script
> > 
> 

> > > Aurora is pretty much a superset of most other generic frameworks sans
> > > maybe
> > > hubspot's singularity.
> > 
> 

> > > On Wednesday, May 13, 2015, Aaron Carey < acarey@ilm.com > wrote:
> > 
> 

> > > I was under the impression Aurora was for long running services? Is it
> > > suitable for scheduling one of batch processes too?
> > 
> 

> > > thanks,
> > 
> 
> > > Aaron
> > 
> 

> > > From: jeffschroed@gmail.com [ jeffschroed@gmail.com ] on behalf of Jeff
> > > Schroeder [ jeffschroeder@computer.org ]
> > 
> 
> > > Sent: 13 May 2015 13:12
> > 
> 
> > > To: user@mesos.apache.org
> > 
> 
> > > Subject: Re: Batch Scheduler with dependency support
> > 
> 

> > > Apache Aurora does this and you can be explicit about the ordering
> > 
> 

> > > On Wednesday, May 13, 2015, Aaron Carey < acarey@ilm.com > wrote:
> > 
> 

> > > Hi All,
> > 
> 

> > > I was just wondering if anyone out there knew of a good mesos batch
> > > scheduler
> > > which supports dependencies between tasks? (ie Task B cannot run until
> > > Task
> > > A is complete)
> > 
> 

> > > Thanks,
> > 
> 
> > > Aaron
> > 
> 

> > > --
> > 
> 
> > > Text by Jeff, typos by iPhone
> > 
> 

> > > --
> > 
> 
> > > Text by Jeff, typos by iPhone
> > 
> 

> > > --
> > 
> 
> > > Text by Jeff, typos by iPhone
> > 
> 

-- 
Cheers, 
Timothy St. Clair 
Red Hat Inc. 

Mime
View raw message