Mailing-List: contact dev-help@systemml.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@systemml.incubator.apache.org
Message-Id: <201604150314.u3F3EtKu027308@d03av04.boulder.ibm.com>
MIME-Version: 1.0
In-Reply-To: <201604150253.u3F2rUCS004743@d03av04.boulder.ibm.com>
Subject: Re: parfor fails
To: dev@systemml.incubator.apache.org
Cc: "Ethan Xu" <ethan.yifanxu@gmail.com>
From: "Matthias Boehm" <mboehm@us.ibm.com>
Date: Thu, 14 Apr 2016 20:14:46 -0700
References: 
 <CAGzHvL=2a==vAAbz=+O1ubCVHKyh_-whn2djz_9qLy4=9u1FMA@mail.gmail.com>
 <201604150253.u3F2rUCS004743@d03av04.boulder.ibm.com>
Content-type: multipart/related;
	Boundary="0__=8FBBF505DF82293E8f9e8a93df938690918c8FBBF505DF82293E"

--0__=8FBBF505DF82293E8f9e8a93df938690918c8FBBF505DF82293E
Content-type: multipart/alternative;
	Boundary="1__=8FBBF505DF82293E8f9e8a93df938690918c8FBBF505DF82293E"


--1__=8FBBF505DF82293E8f9e8a93df938690918c8FBBF505DF82293E
Content-Transfer-Encoding: quoted-printable
Content-type: text/plain; charset=US-ASCII


just for completeness, this issue is tracked with
https://issues.apache.org/jira/browse/SYSTEMML-635 and the fix will be
available tomorrow.

Regards,
Matthias


From:	Matthias Boehm/Almaden/IBM@IBMUS
To:	dev@systemml.incubator.apache.org
Cc:	"Ethan Xu" <ethan.yifanxu@gmail.com>
Date:	04/14/2016 07:53 PM
Subject:	Re: parfor fails


Hi Ethan,

thanks for catching this issue. The parfor script itself is perfectly fine
but you encountered an interesting runtime bug. Usually, you can find the
actual cause at the bottom of the stacktrace or in previous exceptions. I
was able to reproduce this issue if NO systemml config file is provided
(fails on parsing this non-existing config in the parfor mr job task
setup). So the workaround is to put a SystemML-config.xml into the same
directory. Interestingly, the issue did not show up in our testsuite
because we always specify a default configuration there (which was until
recently mandatory).

As a side note, we strongly recommend parfor over for loops here because it
runs the entire loop in 1 instead of 2396 MR jobs due to automatic data
partitioning. However, for the specific example at hand, a data-parallel
formulation (with "s =3D colSums(x=3D=3D0)") would be even better as it all=
ows
for partial aggregation and hence reduces shuffle.

Regards,
Matthias

Ethan Xu ---04/14/2016 01:34:24 PM---Hello, I have a quick question. The
following script fails with this error:

From: Ethan Xu <ethan.yifanxu@gmail.com>
To: dev@systemml.incubator.apache.org
Date: 04/14/2016 01:34 PM
Subject: parfor fails


Hello,

I have a quick question. The following script fails with this error:

org.apache.sysml.runtime.DMLRuntimeException: PARFOR: Failed to execute
loop in parallel.

Here is the dml script:

x=3Dread($X);

print("number of rows of x =3D " + nrow(x));
print("number of cols of x =3D " + ncol(x));

parfor(i in 1:ncol(x), check=3D0){
   a =3D x[,i];
   print("number of 0's in col " + i + " =3D " + sum(a =3D=3D 0));
}

where X is a 35 million by 2396 matrix (coded and dummy coded numerical
matrix) on HDFS. The script runs fine with regular 'for' loops.

Could someone explain why this script cannot run in parallel? Was it a
wrong way to code parfor?

Thanks,

Ethan


--1__=8FBBF505DF82293E8f9e8a93df938690918c8FBBF505DF82293E
Content-Transfer-Encoding: quoted-printable
Content-type: text/html; charset=US-ASCII
Content-Disposition: inline

<html><body><p>just for completeness, this issue is tracked with <a href=3D=
"https://issues.apache.org/jira/browse/SYSTEMML-635">https://issues.apache.=
org/jira/browse/SYSTEMML-635</a> and the fix will be available tomorrow.<br=
><br>Regards,<br>Matthias<br><br><img width=3D"16" height=3D"16" src=3D"cid=
:1=5F=5F=3D8FBBF505DF82293E8f9e8a93df938690918c8FB@" border=3D"0" alt=3D"In=
active hide details for Matthias Boehm---04/14/2016 07:53:43 PM---Hi Ethan,=
 thanks for catching this issue. The parfor script"><font color=3D"#424282"=
>Matthias Boehm---04/14/2016 07:53:43 PM---Hi Ethan, thanks for catching th=
is issue. The parfor script itself is perfectly fine</font><br><br><font si=
ze=3D"2" color=3D"#5F5F5F">From:        </font><font size=3D"2">Matthias Bo=
ehm/Almaden/IBM@IBMUS</font><br><font size=3D"2" color=3D"#5F5F5F">To:     =
   </font><font size=3D"2">dev@systemml.incubator.apache.org</font><br><fon=
t size=3D"2" color=3D"#5F5F5F">Cc:        </font><font size=3D"2">&quot;Eth=
an Xu&quot; &lt;ethan.yifanxu@gmail.com&gt;</font><br><font size=3D"2" colo=
r=3D"#5F5F5F">Date:        </font><font size=3D"2">04/14/2016 07:53 PM</fon=
t><br><font size=3D"2" color=3D"#5F5F5F">Subject:        </font><font size=
=3D"2">Re: parfor fails</font><br><hr width=3D"100%" size=3D"2" align=3D"le=
ft" noshade style=3D"color:#8091A5; "><br><br><br><font size=3D"4">Hi Ethan=
,<br><br>thanks for catching this issue. The parfor script itself is perfec=
tly fine but you encountered an interesting runtime bug. Usually, you can f=
ind the actual cause at the bottom of the stacktrace or in previous excepti=
ons. I was able to reproduce this issue if NO systemml config file is provi=
ded (fails on parsing this non-existing config in the parfor mr job task se=
tup). So the workaround is to put a SystemML-config.xml into the same direc=
tory. Interestingly, the issue did not show up in our testsuite because we =
always specify a default configuration there (which was until recently mand=
atory).<br><br>As a side note, we strongly recommend parfor over for loops =
here because it runs the entire loop in 1 instead of 2396 MR jobs due to au=
tomatic data partitioning. However, for the specific example at hand, a dat=
a-parallel formulation (with &quot;</font><font size=3D"4" face=3D"Lucida C=
onsole">s =3D colSums(x=3D=3D0)</font><font size=3D"4">&quot;) would be eve=
n better as it allows for partial aggregation and hence reduces shuffle.<br=
><br>Regards,<br>Matthias<br><br></font><font size=3D"4" color=3D"#424282">=
Ethan Xu ---04/14/2016 01:34:24 PM---Hello, I have a quick question. The fo=
llowing script fails with this error:</font><font size=3D"4"><br></font><fo=
nt color=3D"#5F5F5F"><br>From: </font>Ethan Xu &lt;ethan.yifanxu@gmail.com&=
gt;<font color=3D"#5F5F5F"><br>To: </font>dev@systemml.incubator.apache.org=
<font color=3D"#5F5F5F"><br>Date: </font>04/14/2016 01:34 PM<font color=3D"=
#5F5F5F"><br>Subject: </font>parfor fails<font size=3D"4"><br></font><hr wi=
dth=3D"100%" size=3D"2" align=3D"left" noshade><font size=3D"4"><br><br></f=
ont><tt><font size=3D"4"><br>Hello,<br><br>I have a quick question. The fol=
lowing script fails with this error:<br><br>org.apache.sysml.runtime.DMLRun=
timeException: PARFOR: Failed to execute<br>loop in parallel.<br><br>Here i=
s the dml script:<br><br>x=3Dread($X);<br><br>print(&quot;number of rows of=
 x =3D &quot; + nrow(x));<br>print(&quot;number of cols of x =3D &quot; + n=
col(x));<br><br>parfor(i in 1:ncol(x), check=3D0){<br> &nbsp; a =3D x[,i];<=
br> &nbsp; print(&quot;number of 0's in col &quot; + i + &quot; =3D &quot; =
+ sum(a =3D=3D 0));<br>}<br><br>where X is a 35 million by 2396 matrix (cod=
ed and dummy coded numerical<br>matrix) on HDFS. The script runs fine with =
regular 'for' loops.<br><br>Could someone explain why this script cannot ru=
n in parallel? Was it a<br>wrong way to code parfor?<br><br>Thanks,<br><br>=
Ethan</font></tt><font size=3D"4"><br><br></font><br><BR>
</body></html>

--1__=8FBBF505DF82293E8f9e8a93df938690918c8FBBF505DF82293E--


--0__=8FBBF505DF82293E8f9e8a93df938690918c8FBBF505DF82293E--