Mailing-List: contact dev-help@systemml.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@systemml.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <CAO1aQR_=KZN5T0oUK4kQ_pcTawwCyvAS9_bRC1ROcuJ2AVd65Q@mail.gmail.com>
Subject: Re: Simplification of MLContext and related APIs
To: dev@systemml.incubator.apache.org
From: "Matthias Boehm" <mboehm@us.ibm.com>
Date: Mon, 12 Sep 2016 23:21:48 +0200
References: <OF01995B75.30DB4100-ON0025802B.005A3810-8825802B.005BD86C@notes.na.collabserv.com>
 <1742075068.3375305.1473641550865@mail.yahoo.com>
 <CAOE9byEktRy1E0Y4EKjWS8YLGbqZBZZkTTQ44XXTjLOnVJu=_g@mail.gmail.com>
 <CAO1aQR_=KZN5T0oUK4kQ_pcTawwCyvAS9_bRC1ROcuJ2AVd65Q@mail.gmail.com>
Content-type: multipart/related;
	Boundary="0__=8FBB0ABFDFE7BB778f9e8a93df938690918c8FBB0ABFDFE7BB77"
Message-Id: <OFFD2E9FA1.89D2A705-ON0025802C.00743DE7-8825802C.00755A65@notes.na.collabserv.com>
archived-at: Mon, 12 Sep 2016 21:22:08 -0000

--0__=8FBB0ABFDFE7BB778f9e8a93df938690918c8FBB0ABFDFE7BB77
Content-type: multipart/alternative;
	Boundary="1__=8FBB0ABFDFE7BB778f9e8a93df938690918c8FBB0ABFDFE7BB77"


--1__=8FBB0ABFDFE7BB778f9e8a93df938690918c8FBB0ABFDFE7BB77
Content-Transfer-Encoding: quoted-printable
Content-type: text/plain; charset=US-ASCII


great - then we're all on the same page. Let me just clarify two aspects:
First, I think we do need abstract frame/matrix data types at API level,
but just one type that is used consistently across MLContext and all DSLs
we're about to add. Second, relying on a common compilation chain does not
directly affect users but ensures consistent behavior across all APIs.

So the bottom line is, we're going to remove MatrixObject/FrameObject and
other internal structures from API level, remove the
BinaryBlockMatrix/BinaryBlockFrame types, and try to consolidate the
various Matrix/Frame objects as well as replicated compilation chains.

Regards,
Matthias


From:	Deron Eriksson <deroneriksson@gmail.com>
To:	dev@systemml.incubator.apache.org
Date:	09/12/2016 01:56 PM
Subject:	Re: Simplification of MLContext and related APIs


Feel free to not expose MatrixObject and FrameObject. I am fine with that.
The only reason MatrixObject and FrameObject are exposed is that I felt if
the new MLContext API did not expose them, there would be complaints from
existing committers that these objects were not available. I can't see
anyone outside of SystemML core developers caring about MatrixObject and
FrameObject or even for that matter ever even using these classes. Users
want DataFrames, DataSets, RDDs, 2D arrays, CSV files, or practically
anything but a MatrixObject or FrameObject.

If you remove entities such as Matrix and Frame, you have the older
MLContext API. Perhaps users who don't wish to use objects such as Matrix
and Frame can use the older API since these suggestions are already built
into the old API?

Deron


On Mon, Sep 12, 2016 at 1:22 PM, Mike Dusenberry <dusenberrymw@gmail.com>
wrote:

> I also agree that internal data structures shouldn't be exposed to a
user.
> However, I think we definitely need to keep the `Matrix` and `Frame`
types
> in the API, in agreement with Arvind.  The main purpose of SystemML for a
> user is to allow for machine learning algorithms involving matrices to be
> run on a given system (laptop, Spark cluster, etc.).  Anything involving
a
> compilation chain directly is noise for our ML users.  Thus it's quite
> useful for SystemML to expose a `Matrix` type with a limited API as is
> currently done in MLContext.  This allows a user to interact with
SystemML
> via these `Matrix` objects which abstractly represent the core data
> structure of a SystemML script.  Furthermore, these Matrix objects can be
> used as subsequent input to an additional script, or can be converted to
a
> DataFrame once the user is ready to continue interacting with Spark.  As
> Arvind mentioned, this just allows the DML `Matrix` type to be
effectively
> exposed at the API level as well.  Additionally, we plan to unify this
> `Matrix` type with the lazy matrix types we are creating in the Python
and
> Scala DSLs, thus allowing `Matrix` to be the equivalent of matrices in
> DML.  The similar argument exists for `Frame` as well.
>
> I think that limiting the exposure of internal structures to users could
be
> useful, but removing `Matrix` & `Frame` and instead having a user deal
> directly with compilation chains would be a step backwards.
>
> - Mike
>
> --
>
> Michael W. Dusenberry
> GitHub: github.com/dusenberrymw
> LinkedIn: linkedin.com/in/mikedusenberry
>
> On Sun, Sep 11, 2016 at 5:52 PM, Acs S <acs=5Fs@yahoo.com.invalid> wrote:
>
> > Yes, I agree that we should NOT expose any internal objects at API
> > level.Objects like FrameObject, MatrixObject should not be exposed as
> those
> > are internal objects.
> > Rule of thumb should be if object (Frame, Object or Scalar) is exposed
at
> > DML level it should be exposed at MlContext level.If there is need to
> > add anything extra object besides being exposed in DML it should be
> > justifiable with rationale.
> > I have introduced FrameObject as oversight. It should have been private
> > method instead of public method. I can fix it soon. But there are more
> > changes you have proposed I will let Deron to respond.
> > Thanks for catching these issues.
> > -Arvind
> >
> >       From: Matthias Boehm <mboehm@us.ibm.com>
> >  To: dev <dev@systemml.incubator.apache.org>
> >  Sent: Sunday, September 11, 2016 9:43 AM
> >  Subject: Simplification of MLContext and related APIs
> >
> >
> >
> > It's great to see the ongoing progress on MLContext and related APIs.
> > However, one aspect that really concerns me is the creation of many
> > redundant data types and exposition of various internal data
structures.
> > For example, exposing MatrixObject and FrameObject at API level is
> > dangerous because it makes external programs data-dependent on internal
> > structures that might be subject to change (no API stability) and users
> > might not be aware of the implications their interactions have on the
> > buffer pool etc. Furthermore, having such a plethora of entry points
> makes
> > it very hard to ensure consistency of the compilation chain with regard
> to
> > configuration handling, environment setup and advanced compilation
> > techniques.
> >
> > I would recommend to create a holistic design across the various APIs
> that
> > aims to (1) reduce the number of exposed data types (for instance, I
> would
> > like to remove MatrixObject/FrameObject from the external interface, as
> > well as remove BinaryBlockMatrix, BinaryBlockFrame, Matrix, Frame, and
> > related meta data objects), and (2) create a configurable compilation
> chain
> > that is invoked from all external APIs. I understand that these data
> types
> > were introduced to simplify, for example, imports in user programs but
> I'm
> > sure we find an alternative realization with less redundancy. What do
you
> > think?
> >
> > Regards,
> > Matthias
> >
> >
> >
>


--1__=8FBB0ABFDFE7BB778f9e8a93df938690918c8FBB0ABFDFE7BB77
Content-Transfer-Encoding: quoted-printable
Content-type: text/html; charset=US-ASCII
Content-Disposition: inline

<html><body><p>great - then we're all on the same page. Let me just clarify=
 two aspects: First, I think we do need abstract frame/matrix data types at=
 API level, but just one type that is used consistently across MLContext an=
d all DSLs we're about to add. Second, relying on a common compilation chai=
n does not directly affect users but ensures consistent behavior across all=
 APIs.<br><br>So the bottom line is, we're going to remove MatrixObject/Fra=
meObject and other internal structures from API level, remove the BinaryBlo=
ckMatrix/BinaryBlockFrame types, and try to consolidate the various Matrix/=
Frame objects as well as replicated compilation chains.<br><br>Regards,<br>=
Matthias<br><br><img width=3D"16" height=3D"16" src=3D"cid:1=5F=5F=3D8FBB0A=
BFDFE7BB778f9e8a93df938690918c8FB@" border=3D"0" alt=3D"Inactive hide detai=
ls for Deron Eriksson ---09/12/2016 01:56:55 PM---Feel free to not expose M=
atrixObject and FrameObject. I am"><font color=3D"#424282">Deron Eriksson -=
--09/12/2016 01:56:55 PM---Feel free to not expose MatrixObject and FrameOb=
ject. I am fine with that. The only reason MatrixObj</font><br><br><font si=
ze=3D"2" color=3D"#5F5F5F">From:        </font><font size=3D"2">Deron Eriks=
son &lt;deroneriksson@gmail.com&gt;</font><br><font size=3D"2" color=3D"#5F=
5F5F">To:        </font><font size=3D"2">dev@systemml.incubator.apache.org<=
/font><br><font size=3D"2" color=3D"#5F5F5F">Date:        </font><font size=
=3D"2">09/12/2016 01:56 PM</font><br><font size=3D"2" color=3D"#5F5F5F">Sub=
ject:        </font><font size=3D"2">Re: Simplification of MLContext and re=
lated APIs</font><br><hr width=3D"100%" size=3D"2" align=3D"left" noshade s=
tyle=3D"color:#8091A5; "><br><br><br><tt>Feel free to not expose MatrixObje=
ct and FrameObject. I am fine with that.<br>The only reason MatrixObject an=
d FrameObject are exposed is that I felt if<br>the new MLContext API did no=
t expose them, there would be complaints from<br>existing committers that t=
hese objects were not available. I can't see<br>anyone outside of SystemML =
core developers caring about MatrixObject and<br>FrameObject or even for th=
at matter ever even using these classes. Users<br>want DataFrames, DataSets=
, RDDs, 2D arrays, CSV files, or practically<br>anything but a MatrixObject=
 or FrameObject.<br><br>If you remove entities such as Matrix and Frame, yo=
u have the older<br>MLContext API. Perhaps users who don't wish to use obje=
cts such as Matrix<br>and Frame can use the older API since these suggestio=
ns are already built<br>into the old API?<br><br>Deron<br><br><br>On Mon, S=
ep 12, 2016 at 1:22 PM, Mike Dusenberry &lt;dusenberrymw@gmail.com&gt;<br>w=
rote:<br><br>&gt; I also agree that internal data structures shouldn't be e=
xposed to a user.<br>&gt; However, I think we definitely need to keep the `=
Matrix` and `Frame` types<br>&gt; in the API, in agreement with Arvind. &nb=
sp;The main purpose of SystemML for a<br>&gt; user is to allow for machine =
learning algorithms involving matrices to be<br>&gt; run on a given system =
(laptop, Spark cluster, etc.). &nbsp;Anything involving a<br>&gt; compilati=
on chain directly is noise for our ML users. &nbsp;Thus it's quite<br>&gt; =
useful for SystemML to expose a `Matrix` type with a limited API as is<br>&=
gt; currently done in MLContext. &nbsp;This allows a user to interact with =
SystemML<br>&gt; via these `Matrix` objects which abstractly represent the =
core data<br>&gt; structure of a SystemML script. &nbsp;Furthermore, these =
Matrix objects can be<br>&gt; used as subsequent input to an additional scr=
ipt, or can be converted to a<br>&gt; DataFrame once the user is ready to c=
ontinue interacting with Spark. &nbsp;As<br>&gt; Arvind mentioned, this jus=
t allows the DML `Matrix` type to be effectively<br>&gt; exposed at the API=
 level as well. &nbsp;Additionally, we plan to unify this<br>&gt; `Matrix` =
type with the lazy matrix types we are creating in the Python and<br>&gt; S=
cala DSLs, thus allowing `Matrix` to be the equivalent of matrices in<br>&g=
t; DML. &nbsp;The similar argument exists for `Frame` as well.<br>&gt;<br>&=
gt; I think that limiting the exposure of internal structures to users coul=
d be<br>&gt; useful, but removing `Matrix` &amp; `Frame` and instead having=
 a user deal<br>&gt; directly with compilation chains would be a step backw=
ards.<br>&gt;<br>&gt; - Mike<br>&gt;<br>&gt; --<br>&gt;<br>&gt; Michael W. =
Dusenberry<br>&gt; GitHub: github.com/dusenberrymw<br>&gt; LinkedIn: linked=
in.com/in/mikedusenberry<br>&gt;<br>&gt; On Sun, Sep 11, 2016 at 5:52 PM, A=
cs S &lt;acs=5Fs@yahoo.com.invalid&gt; wrote:<br>&gt;<br>&gt; &gt; Yes, I a=
gree that we should NOT expose any internal objects at API<br>&gt; &gt; lev=
el.Objects like FrameObject, MatrixObject should not be exposed as<br>&gt; =
those<br>&gt; &gt; are internal objects.<br>&gt; &gt; Rule of thumb should =
be if object (Frame, Object or Scalar) is exposed at<br>&gt; &gt; DML level=
 it should be exposed at MlContext level.If there is need to<br>&gt; &gt; a=
dd anything extra object besides being exposed in DML it should be<br>&gt; =
&gt; justifiable with rationale.<br>&gt; &gt; I have introduced FrameObject=
 as oversight. It should have been private<br>&gt; &gt; method instead of p=
ublic method. I can fix it soon. But there are more<br>&gt; &gt; changes yo=
u have proposed I will let Deron to respond.<br>&gt; &gt; Thanks for catchi=
ng these issues.<br>&gt; &gt; -Arvind<br>&gt; &gt;<br>&gt; &gt; &nbsp; &nbs=
p; &nbsp; From: Matthias Boehm &lt;mboehm@us.ibm.com&gt;<br>&gt; &gt; &nbsp=
;To: dev &lt;dev@systemml.incubator.apache.org&gt;<br>&gt; &gt; &nbsp;Sent:=
 Sunday, September 11, 2016 9:43 AM<br>&gt; &gt; &nbsp;Subject: Simplificat=
ion of MLContext and related APIs<br>&gt; &gt;<br>&gt; &gt;<br>&gt; &gt;<br=
>&gt; &gt; It's great to see the ongoing progress on MLContext and related =
APIs.<br>&gt; &gt; However, one aspect that really concerns me is the creat=
ion of many<br>&gt; &gt; redundant data types and exposition of various int=
ernal data structures.<br>&gt; &gt; For example, exposing MatrixObject and =
FrameObject at API level is<br>&gt; &gt; dangerous because it makes externa=
l programs data-dependent on internal<br>&gt; &gt; structures that might be=
 subject to change (no API stability) and users<br>&gt; &gt; might not be a=
ware of the implications their interactions have on the<br>&gt; &gt; buffer=
 pool etc. Furthermore, having such a plethora of entry points<br>&gt; make=
s<br>&gt; &gt; it very hard to ensure consistency of the compilation chain =
with regard<br>&gt; to<br>&gt; &gt; configuration handling, environment set=
up and advanced compilation<br>&gt; &gt; techniques.<br>&gt; &gt;<br>&gt; &=
gt; I would recommend to create a holistic design across the various APIs<b=
r>&gt; that<br>&gt; &gt; aims to (1) reduce the number of exposed data type=
s (for instance, I<br>&gt; would<br>&gt; &gt; like to remove MatrixObject/F=
rameObject from the external interface, as<br>&gt; &gt; well as remove Bina=
ryBlockMatrix, BinaryBlockFrame, Matrix, Frame, and<br>&gt; &gt; related me=
ta data objects), and (2) create a configurable compilation<br>&gt; chain<b=
r>&gt; &gt; that is invoked from all external APIs. I understand that these=
 data<br>&gt; types<br>&gt; &gt; were introduced to simplify, for example, =
imports in user programs but<br>&gt; I'm<br>&gt; &gt; sure we find an alter=
native realization with less redundancy. What do you<br>&gt; &gt; think?<br=
>&gt; &gt;<br>&gt; &gt; Regards,<br>&gt; &gt; Matthias<br>&gt; &gt;<br>&gt;=
 &gt;<br>&gt; &gt;<br>&gt;<br></tt><br><BR>
</body></html>

--1__=8FBB0ABFDFE7BB778f9e8a93df938690918c8FBB0ABFDFE7BB77--


--0__=8FBB0ABFDFE7BB778f9e8a93df938690918c8FBB0ABFDFE7BB77--