reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saikat Kanjilal <>
Subject Re: [jira] [Commented] (REEF-1791) Implement reef-runtime-spark
Date Mon, 08 May 2017 18:05:44 GMT
Sergei/Markus and other interested folks,
As I'm working on the design for this JIRA I was wondering if we should
include the ability to spin up a spark cluster either on top of mesos
and/or yarn in the user's dev environment for running the reef tasks on the
spark executors, thoughts on this?   This runtime is a little bit different
than mesos or yarn as it actually leverages one of those to run the reef
tasks on the spark executors.  We could in the alternative choose to spin
up the spark cluster locally (meaning not run without any resource manager)
.  The design that I've been thinking is shown pictorially below

reef task-->spark executor-->yarn/mesos/local

The first arrow represents that the reef task is embedded inside the spark
executor and the second is how resource management gets handled


On Tue, May 2, 2017 at 8:49 AM, Saikat Kanjilal (JIRA) <>

>     [
> com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel&focusedCommentId=15993137#comment-15993137 ]
> Saikat Kanjilal commented on REEF-1791:
> ---------------------------------------
> [~motus][~markus.weimer] I will move ahead with my approach based on this
> discussion.  I will
> 1) read through the code I moved over in detail to get a deep
> understanding of the workflow
> 2) Create a design doc on the approach for spark
> 3) what else, get coding :)))) and add unit tests
> On number 3 I noted that a few of the other reef-runtimes don't have unit
> tests, may be something to think about to add to in the future
> More to come as I make more progress, stay tuned.
> > Implement reef-runtime-spark
> > ----------------------------
> >
> >                 Key: REEF-1791
> >                 URL:
> >             Project: REEF
> >          Issue Type: New Feature
> >          Components: REEF
> >            Reporter: Sergiy Matusevych
> >            Assignee: Saikat Kanjilal
> >   Original Estimate: 1,344h
> >  Remaining Estimate: 1,344h
> >
> > We need to run REEF Tasks on Spark Executors. Ideally, that should
> require only a few lines of changes in the REEF application configuration.
> All Spark-related logic must be encapsulated in the {{reef-runtime-spark}}
> module, similar to the existing e.g. {{reef-runtime-yarn}} or
> {{reef-runtime-local}}. As a first step, we can have a Java-only solution,
> but later we'll need to run .NET Tasks on Executors as well.
> --
> This message was sent by Atlassian JIRA
> (v6.3.15#6346)

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message