Mailing-List: contact dev-help@systemml.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@systemml.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <20160524052329.F406B6A054@b03ledav003.gho.boulder.ibm.com>
Subject: Re: Fw: Questions/query about recode / transform in systemML
To: dev@systemml.incubator.apache.org
From: "Matthias Boehm" <mboehm@us.ibm.com>
Date: Tue, 24 May 2016 00:18:35 -0700
References: <20160524052329.F406B6A054@b03ledav003.gho.boulder.ibm.com>
Content-type: multipart/related;
	Boundary="0__=8FBBF52EDFB615008f9e8a93df938690918c8FBBF52EDFB61500"
Message-Id: <20160524071844.7721BBE03E@b03ledav005.gho.boulder.ibm.com>
archived-at: Tue, 24 May 2016 07:18:54 -0000

--0__=8FBBF52EDFB615008f9e8a93df938690918c8FBBF52EDFB61500
Content-type: multipart/alternative;
	Boundary="1__=8FBBF52EDFB615008f9e8a93df938690918c8FBBF52EDFB61500"


--1__=8FBBF52EDFB615008f9e8a93df938690918c8FBBF52EDFB61500
Content-Transfer-Encoding: quoted-printable
Content-type: text/plain; charset=US-ASCII


thanks for the question Alok - couple of comments:

Q1) Independent of the number of columns, we will always do two passes, one
to compute the recode maps (distinct values) and one to apply the recode
maps to your data.

Q2) For the distributed case, we have right now only broadcast-based
transform apply operators. This means it will run out of memory/into errors
if the recode maps do not fit into MR tasks or Spark's broadcast buffers
(2GB because recode maps are not partitioned). However, note that we're
currently in the process of adding native support for frames (see
SYSTEMML-554) - as part of it, we'll also change transform to exploit the
distributed frame representations (SYSTEMML-569), which will already remove
some of the existing restrictions. Further fully distributed transform
operators are certainly possible too (via join-based plans).

Regards,
Matthias


From:	Alok Singh/San Francisco/IBM@IBMUS
To:	dev@systemml.incubator.apache.org
Date:	05/23/2016 10:32 PM
Subject:	Fw: Questions/query about recode / transform in systemML


Hi

Sending it to the dev list as per Matthias suggestions

Alok

----- Forwarded by Alok Singh/San Francisco/IBM on 05/23/2016 10:04 PM
-----

From:   Matthias Boehm/Almaden/IBM
To:     Alok Singh/San Francisco/IBM@IBMUS
Cc:     Arvind Surve/San Jose/IBM@IBMUS
Date:   05/23/2016 09:02 PM
Subject:        Re: Questions/query about recode / transform in systemML


Hi Alok,

would you mind posting this question on our dev mailing list such that
other people also benefit from it? Thanks.


Regards,
Matthias


From:   Alok Singh/San Francisco/IBM
To:     Matthias Boehm/Almaden/IBM@IBMUS, Arvind Surve/San Jose/IBM@IBMUS
Date:   05/23/2016 07:19 PM
Subject:        Questions/query about recode / transform in systemML


Hi Matthias and Arvind.

I had the questions about the internals and how the scan happens in
systemML transform


Question 1

Lets consider an example of dataframe as follows (first line is schema)

userID , county, state
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
1, sanJose,CA
2, santaClara,CA
3,sanJose,CA
4,alameda,CA
5,minnepolis,MN


we can see that uniq for county is {sanJose, alameda, minnepolis} and for
state is {CA,MN}

so example as the doc at
http://apache.github.io/incubator-systemml/files/dml-language-reference/dat=
a.spec.json


user pass in the spec file as
"recode": ["country", "state"]

then the question is how many passes systemML will make for the dataframe
.i.e in general the recode algo would be

for  column  in columns:
   step 1) find uniq for the column

   step 2) apply recode value  for column


so does it mean , we would need 2*count(columns) pass on the dataframe?

if not , then how systemML internally doesn't do more than
2*count(columns)?

Question 2

Lets consider another dataframe as follows (first line is schema)

random=5Fstring
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
col1
dsfsdf
xcvxcv
sdf
etc
foo
Dummy

we can definitely see that number of unique for this df will be almost
same as number of rows
and what if number of rows is 10 trillion and also number of unique for
column random=5Fstring is 10 trillion .
in that case, the whole uniq data will not fit in the one node. so in that
case how does systemML handle it?


Thanks for the inputs
Alok


--1__=8FBBF52EDFB615008f9e8a93df938690918c8FBBF52EDFB61500
Content-Transfer-Encoding: quoted-printable
Content-type: text/html; charset=US-ASCII
Content-Disposition: inline

<html><body><p>thanks for the question Alok - couple of comments:<br><br>Q1=
) Independent of the number of columns, we will always do two passes, one t=
o compute the recode maps (distinct values) and one to apply the recode map=
s to your data. <br><br>Q2) For the distributed case, we have right now onl=
y broadcast-based transform apply operators. This means it will run out of =
memory/into errors if the recode maps do not fit into MR tasks or Spark's b=
roadcast buffers (2GB because recode maps are not partitioned). However, no=
te that we're currently in the process of adding native support for frames =
(see SYSTEMML-554) - as part of it, we'll also change transform to exploit =
the distributed frame representations (SYSTEMML-569), which will already re=
move some of the existing restrictions. Further fully distributed transform=
 operators are certainly possible too (via join-based plans).<br><br>Regard=
s,<br>Matthias<br><br><img width=3D"16" height=3D"16" src=3D"cid:1=5F=5F=3D=
8FBBF52EDFB615008f9e8a93df938690918c8FB@" border=3D"0" alt=3D"Inactive hide=
 details for Alok Singh---05/23/2016 10:32:03 PM---Hi  Sending it to the de=
v list as per Matthias suggestions"><font color=3D"#424282">Alok Singh---05=
/23/2016 10:32:03 PM---Hi  Sending it to the dev list as per Matthias sugge=
stions</font><br><br><font size=3D"2" color=3D"#5F5F5F">From:        </font=
><font size=3D"2">Alok Singh/San Francisco/IBM@IBMUS</font><br><font size=
=3D"2" color=3D"#5F5F5F">To:        </font><font size=3D"2">dev@systemml.in=
cubator.apache.org</font><br><font size=3D"2" color=3D"#5F5F5F">Date:      =
  </font><font size=3D"2">05/23/2016 10:32 PM</font><br><font size=3D"2" co=
lor=3D"#5F5F5F">Subject:        </font><font size=3D"2">Fw: Questions/query=
 about recode / transform in systemML</font><br><hr width=3D"100%" size=3D"=
2" align=3D"left" noshade style=3D"color:#8091A5; "><br><br><br><tt>Hi <br>=
<br>Sending it to the dev list as per Matthias suggestions<br><br>Alok<br><=
br>----- Forwarded by Alok Singh/San Francisco/IBM on 05/23/2016 10:04 PM <=
br>-----<br><br>From: &nbsp; Matthias Boehm/Almaden/IBM<br>To: &nbsp; &nbsp=
; Alok Singh/San Francisco/IBM@IBMUS<br>Cc: &nbsp; &nbsp; Arvind Surve/San =
Jose/IBM@IBMUS<br>Date: &nbsp; 05/23/2016 09:02 PM<br>Subject: &nbsp; &nbsp=
; &nbsp; &nbsp;Re: Questions/query about recode / transform in systemML<br>=
<br><br>Hi Alok,<br><br>would you mind posting this question on our dev mai=
ling list such that <br>other people also benefit from it? Thanks.<br><br><=
br>Regards,<br>Matthias<br><br><br><br>From: &nbsp; Alok Singh/San Francisc=
o/IBM<br>To: &nbsp; &nbsp; Matthias Boehm/Almaden/IBM@IBMUS, Arvind Surve/S=
an Jose/IBM@IBMUS<br>Date: &nbsp; 05/23/2016 07:19 PM<br>Subject: &nbsp; &n=
bsp; &nbsp; &nbsp;Questions/query about recode / transform in systemML<br><=
br><br><br>Hi Matthias and Arvind.<br> <br>I had the questions about the in=
ternals and how the scan happens in <br>systemML transform<br> <br> <br>Que=
stion 1<br> <br>Lets consider an example of dataframe as follows (first lin=
e is schema)<br> <br>userID , county, state<br>=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D<br>1, sanJose,CA<br>2, santaClara,CA<br>3,sanJose,CA<=
br>4,alameda,CA<br>5,minnepolis,MN<br> <br> <br>we can see that uniq for co=
unty is {sanJose, alameda, minnepolis} and for <br>state is {CA,MN}<br> <br=
>so example as the doc at <br></tt><tt><a href=3D"http://apache.github.io/i=
ncubator-systemml/files/dml-language-reference/data.spec.json">http://apach=
e.github.io/incubator-systemml/files/dml-language-reference/data.spec.json<=
/a></tt><tt><br> <br>user pass in the spec file as<br>&quot;recode&quot;: [=
&quot;country&quot;, &quot;state&quot;]<br> <br>then the question is how ma=
ny passes systemML will make for the dataframe<br>.i.e in general the recod=
e algo would be<br> <br>for &nbsp;column &nbsp;in columns:<br> &nbsp; step =
1) find uniq for the column<br> <br> &nbsp; step 2) apply recode value &nbs=
p;for column <br> <br> <br>so does it mean , we would need 2*count(columns)=
 pass on the dataframe?<br> <br>if not , then how systemML internally doesn=
't do more than <br>2*count(columns)?<br> <br>Question 2<br> <br>Lets consi=
der another dataframe as follows (first line is schema)<br> <br>random=5Fst=
ring<br>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br>col1<br>dsfsdf<br>xcvxcv<br>sd=
f<br>etc<br>foo<br>Dummy<br> <br>we can definitely see that number of uniqu=
e for this df will be almost <br>same as number of rows<br>and what if numb=
er of rows is 10 trillion and also number of unique for <br>column random=
=5Fstring is 10 trillion .<br>in that case, the whole uniq data will not fi=
t in the one node. so in that <br>case how does systemML handle it?<br> <br=
> <br>Thanks for the inputs<br>Alok<br><br><br><br></tt><br><BR>
</body></html>

--1__=8FBBF52EDFB615008f9e8a93df938690918c8FBBF52EDFB61500--


--0__=8FBBF52EDFB615008f9e8a93df938690918c8FBBF52EDFB61500--