reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergiy Matusevych (JIRA)" <>
Subject [jira] [Commented] (REEF-1791) Implement reef-runtime-spark
Date Tue, 02 May 2017 02:58:04 GMT


Sergiy Matusevych commented on REEF-1791:

Yes, the dependency is (and should be) restricted to {{reef-runtime-spark}} maven project.

Having it under REEF also means that we can continue using REEF JIRA for issue tracking and
REEF GitHub for pull requests - which simplifies things a lot.

Then I can probably move my Spark+REEF example under, say, {{reef-examples-spark}} module.
This will give us a reference for Spark-related issue tracking, as well as a place to have
Spark-related unit tests.

> Implement reef-runtime-spark
> ----------------------------
>                 Key: REEF-1791
>                 URL:
>             Project: REEF
>          Issue Type: New Feature
>          Components: REEF
>            Reporter: Sergiy Matusevych
>            Assignee: Saikat Kanjilal
>   Original Estimate: 1,344h
>  Remaining Estimate: 1,344h
> We need to run REEF Tasks on Spark Executors. Ideally, that should require only a few
lines of changes in the REEF application configuration. All Spark-related logic must be encapsulated
in the {{reef-runtime-spark}} module, similar to the existing e.g. {{reef-runtime-yarn}} or
{{reef-runtime-local}}. As a first step, we can have a Java-only solution, but later we'll
need to run .NET Tasks on Executors as well.

This message was sent by Atlassian JIRA

View raw message