Mailing-List: contact dev-help@reef.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@reef.apache.org
Date: Thu, 30 Jun 2016 23:14:10 +0000 (UTC)
From: "Byung-Gon Chun (JIRA)" <jira@apache.org>
To: dev@reef.apache.org
Message-ID: <JIRA.12985538.1467246553000.8921.1467328450214@Atlassian.JIRA>
In-Reply-To: <JIRA.12985538.1467246553000@Atlassian.JIRA>
References: <JIRA.12985538.1467246553000@Atlassian.JIRA> <JIRA.12985538.1467246553554@arcas>
Subject: [jira] [Comment Edited] (REEF-1477) Provide a data-centric API for
 stitching REEF jobs together
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 30 Jun 2016 23:14:12 -0000


    [ https://issues.apache.org/jira/browse/REEF-1477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15358022#comment-15358022 ] 

Byung-Gon Chun edited comment on REEF-1477 at 6/30/16 11:13 PM:
----------------------------------------------------------------

Isn't this adding a workflow API? If the job dependency is straightforward, pipeline API?
E.g., Spark pipeline API. http://spark.apache.org/docs/latest/ml-guide.html#main-concepts-in-pipelines


was (Author: bgchun):
Isn't this adding a workflow API?

> Provide a data-centric API for stitching REEF jobs together
> -----------------------------------------------------------
>
>                 Key: REEF-1477
>                 URL: https://issues.apache.org/jira/browse/REEF-1477
>             Project: REEF
>          Issue Type: New Feature
>          Components: REEF.NET
>            Reporter: Joo Seong (Jason) Jeong
>
> The typical flow of using REEF to run machine learning data analytics involves submitting several REEF jobs one at a time, each producing some trained model, intermediate data, or other analysis results. Connecting the jobs together, e.g. using a previously trained model to perform predictions on a test dataset, must be separately managed by the user. For a long series of REEF jobs, this is certainly not desirable -  we would like to be able to stitch a sequence of REEF jobs in a declarative fashion. Moreover, as REEF's name suggests, we should reuse resources for consecutive jobs when possible.
> This can be achieved by providing a data-centric API for running REEF that focuses on the objects instead of REEF program details:
> {code}
> // example
> var trainData = load("hdfs://.../");
> var model = trainData.RunIMRU(jobSpec);
> var testData = load("hdfs://.../");
> var transformedData = testData.ApplyTransform(transform);
> var results = transformedData.RunIMRU(jobSpecAndModel);
> results.Store("hdfs://.../");
> {code}
> Each method call on datasets will start a new REEF job on Evaluators - not necessarily a new Driver - and return an object that can be reused later. Users only need to provide the job spec of each stage and not how the stages get linked with each other. Through this API, constructing a  pipeline of data analytics on REEF will get easier and more intuitive.
> This JIRA will serve as an umbrella for the related issues to provide such an API.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)