reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Saikat Kanjilal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (REEF-1791) Implement reef-runtime-spark
Date Tue, 02 May 2017 15:49:05 GMT

    [ https://issues.apache.org/jira/browse/REEF-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15993137#comment-15993137
] 

Saikat Kanjilal commented on REEF-1791:
---------------------------------------

[~motus][~markus.weimer] I will move ahead with my approach based on this discussion.  I will
1) read through the code I moved over in detail to get a deep understanding of the workflow
2) Create a design doc on the approach for spark
3) what else, get coding :)))) and add unit tests

On number 3 I noted that a few of the other reef-runtimes don't have unit tests, may be something
to think about to add to in the future

More to come as I make more progress, stay tuned.

> Implement reef-runtime-spark
> ----------------------------
>
>                 Key: REEF-1791
>                 URL: https://issues.apache.org/jira/browse/REEF-1791
>             Project: REEF
>          Issue Type: New Feature
>          Components: REEF
>            Reporter: Sergiy Matusevych
>            Assignee: Saikat Kanjilal
>   Original Estimate: 1,344h
>  Remaining Estimate: 1,344h
>
> We need to run REEF Tasks on Spark Executors. Ideally, that should require only a few
lines of changes in the REEF application configuration. All Spark-related logic must be encapsulated
in the {{reef-runtime-spark}} module, similar to the existing e.g. {{reef-runtime-yarn}} or
{{reef-runtime-local}}. As a first step, we can have a Java-only solution, but later we'll
need to run .NET Tasks on Executors as well.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message