reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergiy Matusevych <sergiy.matusev...@gmail.com>
Subject Re: [jira] [Commented] (REEF-1791) Implement reef-runtime-spark
Date Tue, 09 May 2017 19:18:53 GMT
Hi Saikat,

I still don't quite understand what you want to do. Are you suggesting that
when we start a REEF job, the client will automatically spin up a YARN or
Mesos cluster and run the job there?

Thank you,
Sergiy.

On Mon, May 8, 2017 at 4:23 PM, Saikat Kanjilal <sxk1969@gmail.com> wrote:

> Markus/Sergiy,
> That's what I was trying to convey from the diagram, the scenario is that I
> have some job that I'd like to run on a set of spark executors (one can
> imagine a distributed machine learning algorithm), however each of these
> spark executors needs to execute on a cluster with a set of head and worker
> nodes , namely a spark cluster that runs locally in the dev environment ,
> in an integration environment the spark cluster will run in distributed
> mode on yarn or mesos.
>
> Clear?
>
> On Mon, May 8, 2017 at 1:48 PM, Markus Weimer <markus@weimo.de> wrote:
>
> > > As I'm working on the design for this JIRA I was wondering if we should
> > > include the ability to spin up a spark cluster either on top of mesos
> > > and/or yarn in the user's dev environment for running the reef tasks on
> > the
> > > spark executors, thoughts on this?
> >
> > I don't think that is needed. The assumption here is that a user has a
> > Spark job they'd like to launch a REEF job from, not the other way
> > around.
> >
> > Markus
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message