systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias Boehm" <>
Subject Re: Gxuides about running SystemML by spark cluster
Date Sun, 03 Apr 2016 05:08:34 GMT

thanks again for catching, yes the change is in
SystemML head now, so please rebuild SystemML or use one of our nightly
builds ( Thanks.

For running SystemML on Spark, you have multiple options ( Either use
MLContext or spark-submit. Since our documentation does not show many
examples for spark-submit yet, here is a typical command line invocation:

../spark/bin/spark-submit \
      --class org.apache.sysml.api.DMLScript \
      --master yarn-client \
      --num-executors 10 \
      --driver-memory 20g \
      --executor-memory 60g \
      --executor-cores 24 \
      --queue default \
      --conf spark.driver.maxResultSize=0 \
      ./SystemML.jar \
      -f test.dml -stats -exec hybrid_spark -nvargs ...

Everything else is similar to the hadoop invocation. We also provide you a
script that simplifies this configuration:
. Keep in mind that if you want to run in yarn-cluster, you should put the
DML script and potentially SystemML-config into HDFS too.


From:	Wenjie Zhuang <>
Cc:	Matthias Boehm/Almaden/IBM@IBMUS
Date:	04/02/2016 07:50 PM
Subject:	Re: Gxuides about running SystemML by spark cluster


I try to run StepLinearRegDS.dml by spark yarn mode today. And I get the
following result. Is it correct?


Reading X and Y...
Best AIC without any features: 4123.134539784949
Best AIC 4068.2916533784332 achieved with feature: 22
Running linear regression with selected features...
Computing the statistics...
Writing the output matrix...

On Sat, Apr 2, 2016 at 8:37 AM, Wenjie Zhuang <> wrote:

  I am now trying to run experiments about SystemML on spark cluster. Could
  you please share some guides about how to run StepLinearRegDS.dml by
  spark cluster?  The official guide I find is most about hadoop.

  Thanks & Have a good weekend!

  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message