Mailing-List: contact dev-help@systemml.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@systemml.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <OFE1ED9B27.2F59ED6B-ON0025803C.0064498D-8825803C.0064B126@notes.na.collabserv.com>
Subject: Re: Proof of Concept: Embedded Scala DSL
To: dev@systemml.incubator.apache.org
From: "Frederick R Reiss" <frreiss@us.ibm.com>
Date: Wed, 28 Sep 2016 12:00:52 -0700
References: <1ffe92ae6a28438a3c376e207e03a69b@posteo.de>
 <OFFDCD08CE.3D1BCF6F-ON00258038.002983CB-88258038.002CF81E@notes.na.collabserv.com>
 <OF202B7910.686D8410-ON00258038.0060EFEB-88258038.0060F7A7@notes.na.collabserv.com>
 <bbf831387dc2dd2fa11903d113126abc@posteo.de>
 <OFE1ED9B27.2F59ED6B-ON0025803C.0064498D-8825803C.0064B126@notes.na.collabserv.com>
Content-type: multipart/related;
	Boundary="0__=07BB0AAFDFF469608f9e8a93df938690918c07BB0AAFDFF46960"
Message-Id: <OFD7531D09.7AC62481-ON8825803C.0067EFF0-8825803C.0068734F@notes.na.collabserv.com>
archived-at: Wed, 28 Sep 2016 19:01:34 -0000

--0__=07BB0AAFDFF469608f9e8a93df938690918c07BB0AAFDFF46960
Content-type: multipart/alternative;
	Boundary="1__=07BB0AAFDFF469608f9e8a93df938690918c07BB0AAFDFF46960"


--1__=07BB0AAFDFF469608f9e8a93df938690918c07BB0AAFDFF46960
Content-Transfer-Encoding: quoted-printable
Content-type: text/plain; charset=US-ASCII


Maybe I'm missing a subtle point here, but why not refactor the existing
class org.apache.sysml.parser.DMLProgram into our common internal
representation across DSLs? This class is already sufficiently expressive
to represent any DML or PyDML program.

Fred


From:	Niketan Pansare/Almaden/IBM@IBMUS
To:	dev@systemml.incubator.apache.org
Date:	09/28/2016 11:20 AM
Subject:	Re: Proof of Concept: Embedded Scala DSL


Thanks Felix for the response.

+1
>> For the future design I will probably make the Matrix and Vector classes

abstract which allows for different concrete implementations. We could
then have one that is backed directly by SystemML and works similar to
the Python DSL in that it just uses mock operators and builds the DML
string that is then executed using SystemML. That way the deep embedding
would reuse the shallow embedding and we could offer the user to either
use the lazy MatrixType on the Repl or write code inside the macro.

Also, I agree that we can postpone the IR and integration of different DSLs
until the work on parallelize is completed.

Thanks,

Niketan Pansare
IBM Almaden Research Center
E-mail: npansar At us.ibm.com
http://researcher.watson.ibm.com/researcher/view.php?person=3Dus-npansar

fschueler---09/28/2016 10:54:37 AM---Hi Niketan, thanks for your
suggestions! I thought about it a bit and here are my

From: fschueler@posteo.de
To: dev@systemml.incubator.apache.org
Date: 09/28/2016 10:54 AM
Subject: Re: Proof of Concept: Embedded Scala DSL


Hi Niketan,

thanks for your suggestions! I thought about it a bit and here are my
ideas on it:

The IR you are describing is basically already my user facing API. I am
not sure how much sense it makes to have an IR that looks exactly like
the API but with control structures renamed. A common IR for all DSLs
definitely makes sense in general but I am not sure if it should be part
of one particular DSL. For maintainability it might be better to have
that IR somewhere on the SystemML side.

Apart from that and to what Matthias suggested, I thought about how to
make the DSL more suitable for using on the Repl and I think we can find
a good compromise. Currently my API is backed by breeze for rapid
prototyping where breeze just forces evaluation of every statement. For
the future design I will probably make the Matrix and Vector classes
abstract which allows for different concrete implementations. We could
then have one that is backed directly by SystemML and works similar to
the Python DSL in that it just uses mock operators and builds the DML
string that is then executed using SystemML. That way the deep embedding
would reuse the shallow embedding and we could offer the user to either
use the lazy MatrixType on the Repl or write code inside the macro.

I haven't started playing around with this idea but let me know what you
think of it. The lazy, shallow DSL would basically do what you would
want from a seperate IR, but i don't know if you want to call that from
the python DSL.

Felix

Am 24.09.2016 19:39 schrieb Niketan Pansare:
> Hi Felix,
>
> Thanks for the summary. The document is extremely useful. I
> particularly like the idea of parallelizing the code with 'breeze'
> library. I would like to pitch in few ideas which would enable your
> code to be reused by other DSLs:
> 1. Scala DSL/parallelize macro remains the same as described in your
> documentation, but instead of generating DML directly, we call an
> intermediate representation (IR). This IR then generates DML (instead
> of generating DML directly by parallelize). This IR will be then
> reused by Python DSL and R DSL.
> 2. As an example, IR could be a lazy Matrix class (which would be part
> of SystemML). It could have awkward syntax/mechanism for pushing down
> control structures for example: beginWhile and endWhile. Since IR will
> not be exposed to the end-user, it should be fine.
>
> Example:
>
https://github.com/apache/incubator-systemml/blob/master/src/main/python/sy=
stemml/defmatrix.py#L537

> [1] will call IR's add() method. At the end of parallelize or when the
> user wants result (i.e. eval() ), IR could generate DML code and
> execute it.
>
> Again, this is just a proposal and am fine dropping the idea of
> integrating different DSL if it makes the implementation of Scala DSL
> complicated. Also, please feel free to correct me if I am missing
> anything.
>
> Thanks,
>
> Niketan Pansare
> IBM Almaden Research Center
> E-mail: npansar At us.ibm.com
> http://researcher.watson.ibm.com/researcher/view.php?person=3Dus-npansar
> [2]
>
> Matthias Boehm---09/24/2016 01:11:36 AM---thanks for sharing the
> summary - this is very nice. While looking over the example, I had the
> follow
>
> From: Matthias Boehm/Almaden/IBM@IBMUS
> To: dev@systemml.incubator.apache.org
> Date: 09/24/2016 01:11 AM
> Subject: Re: Proof of Concept: Embedded Scala DSL
>
> -------------------------
>
> thanks for sharing the summary - this is very nice. While looking over
> the example, I had the following questions:
>
> 1) Output handling: It would be great to see an example how the
> results of Algorithm.execute() are consumed. Do you intend to hand out
> our binary matrix representation or MLContext's Matrix from which the
> user then requests specific output formats? Also if there are multiple
> Algorithm instances, how is the MLContext (with its internal state of
> lazily evaluated intermediates) reused?
>
> 2) Scala-breeze prototyping: How do you intend to support operations
> that are not supported in breeze? Examples are removeEmpty, table,
> aggregate, rowIndexMax, quantile/centralmoment, cummin/cummax, and DNN
> operations?
>
> 3) Frame data type and operations: Do you also intend to add a frame
> type and its operations? I think for this initial prototype it is not
> necessarily required but please make the scope explicit.
>
> Regards,
> Matthias
>
> fschueler---09/23/2016 04:36:14 PM---As discussed in the related Jira
> (SYSTEMML-451) I have started to implement a prototype/proof of co
>
> From: fschueler@posteo.de
> To: dev@systemml.incubator.apache.org
> Date: 09/23/2016 04:36 PM
> Subject: Proof of Concept: Embedded Scala DSL
>
> -------------------------
>
> As discussed in the related Jira (SYSTEMML-451) I have started to
> implement a prototype/proof of concept for an embedded DSL in Scala.
>
> I have summarized the current approach in a short document that you
> can
> find on github together with the code:
> https://github.com/fschueler/emma/blob/sysml-dsl/emma-sysml-dsl/README.md
> [3]
> Please note that current development happens in the Emma project but
> will move to an independent module in the SystemML project once the
> necessary additions to Emma are merged. By having the DSL in a
> separate
> module, we can include Scala and Emma dependencies only for the users
> that actually want to use the Scala DSL.
>
> The current code serves as a proof of concept to discuss further
> development with the SystemML community. I especially welcome input
> from
> SystemML Scala users on the usability of the API design.
> Next steps will include the translation from Scala code to DML with
> support of all features currently supported in DML, including control
> flow structures.
> Also, a coherent way of executing the generated scripts from Scala and
>
> the interaction with outside data formats (such as Spark Dataframes)
> will be integrated.
>
> I am happy to answer your questions and discuss the described approach
>
> here!
>
> Felix
>
>
>
> Links:
> ------
> [1]
>
https://github.com/apache/incubator-systemml/blob/master/src/main/python/sy=
stemml/defmatrix.py#L537

> [2]
> http://researcher.watson.ibm.com/researcher/view.php?person=3Dus-npansar
> [3]
> https://github.com/fschueler/emma/blob/sysml-dsl/emma-sysml-dsl/README.md


--1__=07BB0AAFDFF469608f9e8a93df938690918c07BB0AAFDFF46960
Content-Transfer-Encoding: quoted-printable
Content-type: text/html; charset=US-ASCII
Content-Disposition: inline

<html><body><p>Maybe I'm missing a subtle point here, but why not refactor =
the existing class org.apache.sysml.parser.DMLProgram into our common inter=
nal representation across DSLs? This class is already sufficiently expressi=
ve to represent any DML or PyDML program.<br><br>Fred<br><br><img width=3D"=
16" height=3D"16" src=3D"cid:1=5F=5F=3D07BB0AAFDFF469608f9e8a93df938690918c=
07B@" border=3D"0" alt=3D"Inactive hide details for Niketan Pansare---09/28=
/2016 11:20:11 AM---Thanks Felix for the response. +1"><font color=3D"#4242=
82">Niketan Pansare---09/28/2016 11:20:11 AM---Thanks Felix for the respons=
e. +1</font><br><br><font size=3D"2" color=3D"#5F5F5F">From:        </font>=
<font size=3D"2">Niketan Pansare/Almaden/IBM@IBMUS</font><br><font size=3D"=
2" color=3D"#5F5F5F">To:        </font><font size=3D"2">dev@systemml.incuba=
tor.apache.org</font><br><font size=3D"2" color=3D"#5F5F5F">Date:        </=
font><font size=3D"2">09/28/2016 11:20 AM</font><br><font size=3D"2" color=
=3D"#5F5F5F">Subject:        </font><font size=3D"2">Re: Proof of Concept: =
Embedded Scala DSL</font><br><hr width=3D"100%" size=3D"2" align=3D"left" n=
oshade style=3D"color:#8091A5; "><br><br><br><font size=3D"4">Thanks Felix =
for the response.<br><br>+1 <br>&gt;&gt; </font><tt><font size=3D"4">For th=
e future design I will probably make the Matrix and Vector classes <br>abst=
ract which allows for different concrete implementations. We could <br>then=
 have one that is backed directly by SystemML and works similar to <br>the =
Python DSL in that it just uses mock operators and builds the DML <br>strin=
g that is then executed using SystemML. That way the deep embedding <br>wou=
ld reuse the shallow embedding and we could offer the user to either <br>us=
e the lazy MatrixType on the Repl or write code inside the macro.</font></t=
t><font size=3D"4"><br><br>Also, I agree that we can postpone the IR and in=
tegration of different DSLs until the work on parallelize is completed.<br>=
<br>Thanks,<br><br>Niketan Pansare<br>IBM Almaden Research Center<br>E-mail=
: npansar At us.ibm.com</font><u><font size=3D"4" color=3D"#0000FF"><br></f=
ont></u><a href=3D"http://researcher.watson.ibm.com/researcher/view.php?per=
son=3Dus-npansar"><u><font size=3D"4" color=3D"#0000FF">http://researcher.w=
atson.ibm.com/researcher/view.php?person=3Dus-npansar</font></u></a><font s=
ize=3D"4"><br><br></font><font size=3D"4" color=3D"#424282">fschueler---09/=
28/2016 10:54:37 AM---Hi Niketan, thanks for your suggestions! I thought ab=
out it a bit and here are my</font><font size=3D"4"><br></font><font color=
=3D"#5F5F5F"><br>From: </font>fschueler@posteo.de<font color=3D"#5F5F5F"><b=
r>To: </font>dev@systemml.incubator.apache.org<font color=3D"#5F5F5F"><br>D=
ate: </font>09/28/2016 10:54 AM<font color=3D"#5F5F5F"><br>Subject: </font>=
Re: Proof of Concept: Embedded Scala DSL<font size=3D"4"><br></font><hr wid=
th=3D"100%" size=3D"2" align=3D"left" noshade><font size=3D"4"><br><br></fo=
nt><tt><font size=3D"4"><br>Hi Niketan,<br><br>thanks for your suggestions!=
 I thought about it a bit and here are my <br>ideas on it:<br><br>The IR yo=
u are describing is basically already my user facing API. I am <br>not sure=
 how much sense it makes to have an IR that looks exactly like <br>the API =
but with control structures renamed. A common IR for all DSLs <br>definitel=
y makes sense in general but I am not sure if it should be part <br>of one =
particular DSL. For maintainability it might be better to have <br>that IR =
somewhere on the SystemML side.<br><br>Apart from that and to what Matthias=
 suggested, I thought about how to <br>make the DSL more suitable for using=
 on the Repl and I think we can find <br>a good compromise. Currently my AP=
I is backed by breeze for rapid <br>prototyping where breeze just forces ev=
aluation of every statement. For <br>the future design I will probably make=
 the Matrix and Vector classes <br>abstract which allows for different conc=
rete implementations. We could <br>then have one that is backed directly by=
 SystemML and works similar to <br>the Python DSL in that it just uses mock=
 operators and builds the DML <br>string that is then executed using System=
ML. That way the deep embedding <br>would reuse the shallow embedding and w=
e could offer the user to either <br>use the lazy MatrixType on the Repl or=
 write code inside the macro.<br><br>I haven't started playing around with =
this idea but let me know what you <br>think of it. The lazy, shallow DSL w=
ould basically do what you would <br>want from a seperate IR, but i don't k=
now if you want to call that from <br>the python DSL.<br><br>Felix<br><br>A=
m 24.09.2016 19:39 schrieb Niketan Pansare:<br>&gt; Hi Felix,<br>&gt; <br>&=
gt; Thanks for the summary. The document is extremely useful. I<br>&gt; par=
ticularly like the idea of parallelizing the code with 'breeze'<br>&gt; lib=
rary. I would like to pitch in few ideas which would enable your<br>&gt; co=
de to be reused by other DSLs:<br>&gt; 1. Scala DSL/parallelize macro remai=
ns the same as described in your<br>&gt; documentation, but instead of gene=
rating DML directly, we call an<br>&gt; intermediate representation (IR). T=
his IR then generates DML (instead<br>&gt; of generating DML directly by pa=
rallelize). This IR will be then<br>&gt; reused by Python DSL and R DSL.<br=
>&gt; 2. As an example, IR could be a lazy Matrix class (which would be par=
t<br>&gt; of SystemML). It could have awkward syntax/mechanism for pushing =
down<br>&gt; control structures for example: beginWhile and endWhile. Since=
 IR will<br>&gt; not be exposed to the end-user, it should be fine.<br>&gt;=
 <br>&gt; Example:<br>&gt; </font></tt><a href=3D"https://github.com/apache=
/incubator-systemml/blob/master/src/main/python/systemml/defmatrix.py#L537"=
><tt><u><font size=3D"4" color=3D"#0000FF">https://github.com/apache/incuba=
tor-systemml/blob/master/src/main/python/systemml/defmatrix.py#L537</font><=
/u></tt></a><tt><font size=3D"4"><br>&gt; [1] will call IR's add() method. =
At the end of parallelize or when the<br>&gt; user wants result (i.e. eval(=
) ), IR could generate DML code and<br>&gt; execute it.<br>&gt; <br>&gt; Ag=
ain, this is just a proposal and am fine dropping the idea of<br>&gt; integ=
rating different DSL if it makes the implementation of Scala DSL<br>&gt; co=
mplicated. Also, please feel free to correct me if I am missing<br>&gt; any=
thing.<br>&gt; <br>&gt; Thanks,<br>&gt; <br>&gt; Niketan Pansare<br>&gt; IB=
M Almaden Research Center<br>&gt; E-mail: npansar At us.ibm.com<br>&gt; </f=
ont></tt><a href=3D"http://researcher.watson.ibm.com/researcher/view.php?pe=
rson=3Dus-npansar"><tt><u><font size=3D"4" color=3D"#0000FF">http://researc=
her.watson.ibm.com/researcher/view.php?person=3Dus-npansar</font></u></tt><=
/a><tt><font size=3D"4"><br>&gt; [2]<br>&gt; <br>&gt; Matthias Boehm---09/2=
4/2016 01:11:36 AM---thanks for sharing the<br>&gt; summary - this is very =
nice. While looking over the example, I had the<br>&gt; follow<br>&gt; <br>=
&gt; From: Matthias Boehm/Almaden/IBM@IBMUS<br>&gt; To: dev@systemml.incuba=
tor.apache.org<br>&gt; Date: 09/24/2016 01:11 AM<br>&gt; Subject: Re: Proof=
 of Concept: Embedded Scala DSL<br>&gt; <br>&gt; -------------------------<=
br>&gt; <br>&gt; thanks for sharing the summary - this is very nice. While =
looking over<br>&gt; the example, I had the following questions:<br>&gt; <b=
r>&gt; 1) Output handling: It would be great to see an example how the<br>&=
gt; results of Algorithm.execute() are consumed. Do you intend to hand out<=
br>&gt; our binary matrix representation or MLContext's Matrix from which t=
he<br>&gt; user then requests specific output formats? Also if there are mu=
ltiple<br>&gt; Algorithm instances, how is the MLContext (with its internal=
 state of<br>&gt; lazily evaluated intermediates) reused?<br>&gt; <br>&gt; =
2) Scala-breeze prototyping: How do you intend to support operations<br>&gt=
; that are not supported in breeze? Examples are removeEmpty, table,<br>&gt=
; aggregate, rowIndexMax, quantile/centralmoment, cummin/cummax, and DNN<br=
>&gt; operations?<br>&gt; <br>&gt; 3) Frame data type and operations: Do yo=
u also intend to add a frame<br>&gt; type and its operations? I think for t=
his initial prototype it is not<br>&gt; necessarily required but please mak=
e the scope explicit.<br>&gt; <br>&gt; Regards,<br>&gt; Matthias<br>&gt; <b=
r>&gt; fschueler---09/23/2016 04:36:14 PM---As discussed in the related Jir=
a<br>&gt; (SYSTEMML-451) I have started to implement a prototype/proof of c=
o<br>&gt; <br>&gt; From: fschueler@posteo.de<br>&gt; To: dev@systemml.incub=
ator.apache.org<br>&gt; Date: 09/23/2016 04:36 PM<br>&gt; Subject: Proof of=
 Concept: Embedded Scala DSL<br>&gt; <br>&gt; -------------------------<br>=
&gt; <br>&gt; As discussed in the related Jira (SYSTEMML-451) I have starte=
d to<br>&gt; implement a prototype/proof of concept for an embedded DSL in =
Scala.<br>&gt; <br>&gt; I have summarized the current approach in a short d=
ocument that you<br>&gt; can<br>&gt; find on github together with the code:=
<br>&gt; </font></tt><a href=3D"https://github.com/fschueler/emma/blob/sysm=
l-dsl/emma-sysml-dsl/README.md"><tt><u><font size=3D"4" color=3D"#0000FF">h=
ttps://github.com/fschueler/emma/blob/sysml-dsl/emma-sysml-dsl/README.md</f=
ont></u></tt></a><tt><font size=3D"4"><br>&gt; [3]<br>&gt; Please note that=
 current development happens in the Emma project but<br>&gt; will move to a=
n independent module in the SystemML project once the<br>&gt; necessary add=
itions to Emma are merged. By having the DSL in a<br>&gt; separate<br>&gt; =
module, we can include Scala and Emma dependencies only for the users<br>&g=
t; that actually want to use the Scala DSL.<br>&gt; <br>&gt; The current co=
de serves as a proof of concept to discuss further<br>&gt; development with=
 the SystemML community. I especially welcome input<br>&gt; from<br>&gt; Sy=
stemML Scala users on the usability of the API design.<br>&gt; Next steps w=
ill include the translation from Scala code to DML with<br>&gt; support of =
all features currently supported in DML, including control<br>&gt; flow str=
uctures.<br>&gt; Also, a coherent way of executing the generated scripts fr=
om Scala and<br>&gt; <br>&gt; the interaction with outside data formats (su=
ch as Spark Dataframes)<br>&gt; will be integrated.<br>&gt; <br>&gt; I am h=
appy to answer your questions and discuss the described approach<br>&gt; <b=
r>&gt; here!<br>&gt; <br>&gt; Felix<br>&gt; <br>&gt; <br>&gt; <br>&gt; Link=
s:<br>&gt; ------<br>&gt; [1]<br>&gt; </font></tt><a href=3D"https://github=
.com/apache/incubator-systemml/blob/master/src/main/python/systemml/defmatr=
ix.py#L537"><tt><u><font size=3D"4" color=3D"#0000FF">https://github.com/ap=
ache/incubator-systemml/blob/master/src/main/python/systemml/defmatrix.py#L=
537</font></u></tt></a><tt><font size=3D"4"><br>&gt; [2] <br>&gt; </font></=
tt><a href=3D"http://researcher.watson.ibm.com/researcher/view.php?person=
=3Dus-npansar"><tt><u><font size=3D"4" color=3D"#0000FF">http://researcher.=
watson.ibm.com/researcher/view.php?person=3Dus-npansar</font></u></tt></a><=
tt><font size=3D"4"><br>&gt; [3] <br>&gt; </font></tt><a href=3D"https://gi=
thub.com/fschueler/emma/blob/sysml-dsl/emma-sysml-dsl/README.md"><tt><u><fo=
nt size=3D"4" color=3D"#0000FF">https://github.com/fschueler/emma/blob/sysm=
l-dsl/emma-sysml-dsl/README.md</font></u></tt></a><tt><font size=3D"4"><br>=
</font></tt><font size=3D"4"><br><br><br></font><br><br><BR>
</body></html>

--1__=07BB0AAFDFF469608f9e8a93df938690918c07BB0AAFDFF46960--


--0__=07BB0AAFDFF469608f9e8a93df938690918c07BB0AAFDFF46960--