reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saikat Kanjilal <>
Subject Re: [jira] [Commented] (REEF-1791) Implement reef-runtime-spark
Date Mon, 08 May 2017 23:23:46 GMT
That's what I was trying to convey from the diagram, the scenario is that I
have some job that I'd like to run on a set of spark executors (one can
imagine a distributed machine learning algorithm), however each of these
spark executors needs to execute on a cluster with a set of head and worker
nodes , namely a spark cluster that runs locally in the dev environment ,
in an integration environment the spark cluster will run in distributed
mode on yarn or mesos.


On Mon, May 8, 2017 at 1:48 PM, Markus Weimer <> wrote:

> > As I'm working on the design for this JIRA I was wondering if we should
> > include the ability to spin up a spark cluster either on top of mesos
> > and/or yarn in the user's dev environment for running the reef tasks on
> the
> > spark executors, thoughts on this?
> I don't think that is needed. The assumption here is that a user has a
> Spark job they'd like to launch a REEF job from, not the other way
> around.
> Markus

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message