Return-Path: X-Original-To: apmail-systemml-dev-archive@minotaur.apache.org Delivered-To: apmail-systemml-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9353E18AB4 for ; Sun, 3 Apr 2016 05:09:01 +0000 (UTC) Received: (qmail 87096 invoked by uid 500); 3 Apr 2016 05:09:00 -0000 Delivered-To: apmail-systemml-dev-archive@systemml.apache.org Received: (qmail 87044 invoked by uid 500); 3 Apr 2016 05:09:00 -0000 Mailing-List: contact dev-help@systemml.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@systemml.incubator.apache.org Delivered-To: mailing list dev@systemml.incubator.apache.org Received: (qmail 87033 invoked by uid 99); 3 Apr 2016 05:08:59 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 03 Apr 2016 05:08:59 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 6660CC0C99 for ; Sun, 3 Apr 2016 05:08:59 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -3.02 X-Spam-Level: X-Spam-Status: No, score=-3.02 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, TVD_FW_GRAPHIC_NAME_MID=0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id pC9HBPVUkrj7 for ; Sun, 3 Apr 2016 05:08:57 +0000 (UTC) Received: from e33.co.us.ibm.com (e33.co.us.ibm.com [32.97.110.151]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 1384B5F472 for ; Sun, 3 Apr 2016 05:08:55 +0000 (UTC) Received: from localhost by e33.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Sat, 2 Apr 2016 23:08:53 -0600 Received: from d03dlp02.boulder.ibm.com (9.17.202.178) by e33.co.us.ibm.com (192.168.1.133) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Sat, 2 Apr 2016 23:08:44 -0600 X-IBM-Helo: d03dlp02.boulder.ibm.com X-IBM-MailFrom: mboehm@us.ibm.com X-IBM-RcptTo: dev@systemml.incubator.apache.org Received: from b01cxnp23033.gho.pok.ibm.com (b01cxnp23033.gho.pok.ibm.com [9.57.198.28]) by d03dlp02.boulder.ibm.com (Postfix) with ESMTP id 1B2823E4001C for ; Sat, 2 Apr 2016 23:08:44 -0600 (MDT) Received: from d01av02.pok.ibm.com (d01av02.pok.ibm.com [9.56.224.216]) by b01cxnp23033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id u3358hBA41287778 for ; Sun, 3 Apr 2016 05:08:43 GMT Received: from d01av02.pok.ibm.com (localhost [127.0.0.1]) by d01av02.pok.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id u3358h7n029889 for ; Sun, 3 Apr 2016 01:08:43 -0400 Received: from d50lp31.co.us.ibm.com (d50lp31.boulder.ibm.com [9.17.249.32]) by d01av02.pok.ibm.com (8.14.4/8.14.4/NCO v10.0 AVin) with ESMTP id u3358gvx029852 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL) for ; Sun, 3 Apr 2016 01:08:43 -0400 Message-Id: <201604030508.u3358gvx029852@d01av02.pok.ibm.com> Received: from localhost by d50lp31.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Sat, 2 Apr 2016 23:08:42 -0600 Received: from smtp.notes.na.collabserv.com (192.155.248.91) by d50lp31.co.us.ibm.com (192.168.2.141) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256/256) Sat, 2 Apr 2016 23:08:39 -0600 X-IBM-Helo: smtp.notes.na.collabserv.com X-IBM-MailFrom: mboehm@us.ibm.com X-IBM-RcptTo: dev@systemml.incubator.apache.org Received: from /spool/local by smtp.notes.na.collabserv.com with smtp.notes.na.collabserv.com ESMTP for from ; Sun, 3 Apr 2016 05:08:38 -0000 Received: from us1a3-smtp05.a3.dal06.isc4sb.com (10.146.71.159) by smtp.notes.na.collabserv.com (10.106.227.143) with smtp.notes.na.collabserv.com ESMTP; Sun, 3 Apr 2016 05:08:37 -0000 Received: from us1a3-mail149.a3.dal06.isc4sb.com ([10.146.38.84]) by us1a3-smtp05.a3.dal06.isc4sb.com with ESMTP id 2016040305083626-5815 ; Sun, 3 Apr 2016 05:08:36 +0000 MIME-Version: 1.0 In-Reply-To: Subject: Re: Gxuides about running SystemML by spark cluster To: Wenjie Zhuang Cc: dev@systemml.incubator.apache.org From: "Matthias Boehm" Date: Sat, 2 Apr 2016 21:08:34 -0800 References: X-KeepSent: D64AFEB5:9C55D42B-00257F8A:001AFF7B; type=4; name=$KeepSent X-Mailer: IBM Notes Release 9.0.1FP2 SHF37 August 25, 2014 X-LLNOutbound: False X-Disclaimed: 48343 X-TNEFEvaluated: 1 Content-type: multipart/related; Boundary="0__=8FBBF519DF8979EB8f9e8a93df938690918c8FBBF519DF8979EB" x-cbid: 16040305-0009-0000-0000-00001A410932 X-IBM-ISS-SpamDetectors: Score=0.406879; BY=0.233078; FL=0; FP=0; FZ=0; HX=0; KW=0; PH=0; SC=0.406879; ST=0; TS=0; UL=0; ISC= X-IBM-ISS-DetailInfo: BY=3.00005111; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000156; SDB=6.00682554; UDB=6.00313456; UTC=2016-04-03 05:08:38 x-cbparentid: 16040305-4778-0000-0000-000002C63773 X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER --0__=8FBBF519DF8979EB8f9e8a93df938690918c8FBBF519DF8979EB Content-type: multipart/alternative; Boundary="1__=8FBBF519DF8979EB8f9e8a93df938690918c8FBBF519DF8979EB" --1__=8FBBF519DF8979EB8f9e8a93df938690918c8FBBF519DF8979EB Content-Transfer-Encoding: quoted-printable Content-type: text/plain; charset=ISO-8859-1 thanks again for catching https://issues.apache.org/jira/browse/SYSTEMML-609, yes the change is in SystemML head now, so please rebuild SystemML or use one of our nightly builds (https://sparktc.ibmcloud.com/repo/latest/). Thanks. For running SystemML on Spark, you have multiple options ( http://apache.github.io/incubator-systemml/#running-systemml). Either use MLContext or spark-submit. Since our documentation does not show many examples for spark-submit yet, here is a typical command line invocation: ../spark/bin/spark-submit \ --class org.apache.sysml.api.DMLScript \ --master yarn-client \ --num-executors 10 \ --driver-memory 20g \ --executor-memory 60g \ --executor-cores 24 \ --queue default \ --conf spark.driver.maxResultSize=3D0 \ ./SystemML.jar \ -f test.dml -stats -exec hybrid=5Fspark -nvargs ... Everything else is similar to the hadoop invocation. We also provide you a script that simplifies this configuration: https://github.com/apache/incubator-systemml/blob/master/scripts/sparkDML.sh . Keep in mind that if you want to run in yarn-cluster, you should put the DML script and potentially SystemML-config into HDFS too. Regards, Matthias From: Wenjie Zhuang To: dev@systemml.incubator.apache.org Cc: Matthias Boehm/Almaden/IBM@IBMUS Date: 04/02/2016 07:50 PM Subject: Re: Gxuides about running SystemML by spark cluster Hi, I try to run StepLinearRegDS.dml by spark yarn mode today. And I get the following result. Is it correct? Thanks. BEGIN STEPWISE LINEAR REGRESSION SCRIPT Reading X and Y... Best AIC without any features: 4123.134539784949 Best AIC 4068.2916533784332 achieved with feature: 22 Running linear regression with selected features... Computing the statistics... Writing the output matrix... On Sat, Apr 2, 2016 at 8:37 AM, Wenjie Zhuang wrote: Hi, I am now trying to run experiments about SystemML on spark cluster. Could you please share some guides about how to run StepLinearRegDS.dml by spark cluster?=A0 The official guide I find is most about hadoop. Thanks & Have a good weekend! --1__=8FBBF519DF8979EB8f9e8a93df938690918c8FBBF519DF8979EB Content-Transfer-Encoding: quoted-printable Content-type: text/html; charset=ISO-8859-1 Content-Disposition: inline

thanks again for catching https://issues.apache.org/jira/browse/SYSTEMML= -609, yes the change is in SystemML head now, so please rebuild SystemM= L or use one of our nightly builds (https://sparktc.ibmcloud.com/repo/latest/). Thanks.
<= br>For running SystemML on Spark, you have multiple options (http://apache.gi= thub.io/incubator-systemml/#running-systemml). Either use MLContext or = spark-submit. Since our documentation does not show many examples for spark= -submit yet, here is a typical command line invocation:

../spark/bin/spark-submit \
--class org.apache.sysml.api.DMLScript \
--master yarn-client \
--num-executors 10 \
--driver-memory 20g \
--executor-memory 60g \
= --executor-cores 24 \
-= -queue default \
--conf spark= .driver.maxResultSize=3D0 \
.= /SystemML.jar \
-f test.dml -= stats -exec hybrid=5Fspark -nvargs ...

Everything else is si= milar to the hadoop invocation. We also provide you a script that simplifie= s this configuration: https://github.com/apache/incubator-sys= temml/blob/master/scripts/sparkDML.sh. Keep in mind that if you want to= run in yarn-cluster, you should put the DML script and potentially SystemM= L-config into HDFS too.

Regards,
Matthias


3D"InactiveWenjie Zhuang ---04/02/2016 07:50:35 PM---= Hi, I try to run StepLinearRegDS.dml by spark yarn mode today. And I get th= e

From: Wenjie Zhuang <kaito@vt.edu>
To: dev@systemml.incubator= .apache.org
Cc: = Matthias Boehm/Almaden/IBM@IBMUS
Date: 04/02/2016 07:50 P= M
Subject: Re: Gxuides about running SystemML by spark cluster
<= hr width=3D"100%" size=3D"2" align=3D"left" noshade style=3D"color:#8091A5;= ">


Hi,

I try to= run StepLinearRegDS.dml by spark yarn mode today. And I get the following = result. Is it correct?

Thanks.

BEGIN STEPWISE LINEAR REGRESSION SCRIPT
Reading X and = Y...
Best AIC without any features: 4123.134539784949
Best AIC 4068.2= 916533784332 achieved with feature: 22
Running linear regression with se= lected features...
Computing the statistics...
Writing the output mat= rix...



On Sat, Apr 2, 2016 at 8:37 A= M, Wenjie Zhuang <kaito@vt.edu> wrote:

    Hi,

    I am now trying to run experiments about SystemML on spark clus= ter. Could you please share some guides about how to run StepLinearRegDS.dm= l by spark cluster?=A0 The official guide I find is most about hadoop.
    <= /font>
    Thanks & Have a good weekend!


    --1__=8FBBF519DF8979EB8f9e8a93df938690918c8FBBF519DF8979EB-- --0__=8FBBF519DF8979EB8f9e8a93df938690918c8FBBF519DF8979EB--