systemml-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Niketan Pansare (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SYSTEMML-1471) Support PreparedScript for MLContext
Date Thu, 06 Apr 2017 23:53:41 GMT

    [ https://issues.apache.org/jira/browse/SYSTEMML-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15959993#comment-15959993
] 

Niketan Pansare commented on SYSTEMML-1471:
-------------------------------------------

[~deron] Thanks for your reply. Is MLContext thread-safe ? Do you think there are other issues
like multiple (i.e. above script) and/or parallel (i.e. parallel execute calls) invocation
that also needs to be addressed ?

Additionally, we should also consider removing code-duplication in our JMLC and MLContext
APIs if possible.

> Support PreparedScript for MLContext
> ------------------------------------
>
>                 Key: SYSTEMML-1471
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1471
>             Project: SystemML
>          Issue Type: Improvement
>            Reporter: Niketan Pansare
>
> The intent of this JIRA is three-fold:
> 1. Allow MLContext to be used in prediction scenario.
> 2. Consolidate the code of JMLC and MLContext.
> 3. Explore what extensions are needed in SystemML to support Spark streaming.
> For prediction scenario, it is important to reduce the parsing/validation overhead as
much as possible and reusing the JMLC infrastructure might be a good step in that direction.
It is also important that MLContext continues to support dynamic recompilation and other optimization
as the input size could be small (similar to JMLC), but could also be large (if window size
is large, making MLContext ideal for this scenario). 
> {code}
> val streamingContext = new StreamingContext(sc, SLIDE_INTERVAL)
> val windowDStream  = .....window(WINDOW_LENGTH, SLIDE_INTERVAL)
> val preparedScript = ....prepareScript(....)
> windowDStream.foreachRDD(currentWindow => {
> if (currentWindow.count() > 0) {
>   ml.execute(preparedScript.in("X", currentWindow.toDF()))
>   ...
> }
> })
> {code}
> [~deron] [~mboehm7] [~reinwald] [~freiss] [~mwdusenb@us.ibm.com] [~nakul02] Is this something
that interest anyone of you ?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message