reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Saikat Kanjilal (JIRA)" <>
Subject [jira] [Commented] (REEF-1791) Implement reef-runtime-spark
Date Mon, 15 May 2017 16:05:04 GMT


Saikat Kanjilal commented on REEF-1791:

@minterlandi , thanks for the feedback if you look at the livy interface it is the most lightweight
option as it requires no bindings to the spark runtime, the second option I identified is
similar to what you are describing although I'm not sure what you mean by mapPartition because
Option 2 abstracts this away from us, if we can make the assumption that a set of spark executors
are readily available to us then we can just pass the reef task into the set of executors
at which point the master node will manage partitioning the data and doing the work necessary
for the algorithm.    I will look at TensorFlowOnSpark but I suspect its architecture may
be a different than ours although the goal is conceivably the same.

> Implement reef-runtime-spark
> ----------------------------
>                 Key: REEF-1791
>                 URL:
>             Project: REEF
>          Issue Type: New Feature
>          Components: REEF
>            Reporter: Sergiy Matusevych
>            Assignee: Saikat Kanjilal
>   Original Estimate: 1,344h
>  Remaining Estimate: 1,344h
> We need to run REEF Tasks on Spark Executors. Ideally, that should require only a few
lines of changes in the REEF application configuration. All Spark-related logic must be encapsulated
in the {{reef-runtime-spark}} module, similar to the existing e.g. {{reef-runtime-yarn}} or
{{reef-runtime-local}}. As a first step, we can have a Java-only solution, but later we'll
need to run .NET Tasks on Executors as well.

This message was sent by Atlassian JIRA

View raw message