systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Glenn Weidner" <gweid...@us.ibm.com>
Subject Re: NPE running GLM-predict.dml in spark mode
Date Sun, 25 Sep 2016 18:07:32 GMT

This has been resolved in
https://issues.apache.org/jira/browse/SYSTEMML-955.

Thanks,
Glenn



From:	Glenn Weidner/Silicon Valley/IBM@IBMUS
To:	dev@systemml.incubator.apache.org
Date:	09/02/2016 04:00 PM
Subject:	NPE running GLM-predict.dml in spark mode





When running Linear Regression example described at
https://apache.github.io/incubator-systemml/standalone-guide.html#linear-regression-example

, everything works well with spark-submit under hybrid_spark mode.
However, when running the very last step GLM-predict with -exec spark, a
null pointer exception occurs.

spark-submit --master yarn --deploy-mode client --class
org.apache.sysml.api.DMLScript systemml.jar -f GLM-predict.dml -exec spark
-nvargs X=linRegData.test.data.csv Y=linRegData.test.labels.csv B=betas.csv
fmt=csv

16/09/02 15:35:10 ERROR DMLScript: Failed to execute DML script.
org.apache.sysml.lops.LopsException: ERROR: line 240, column 30 -- Problem
generating simple inst - SPARK°cdf°dist=normal°target=¶_Var237¶°
_Var238·SCALAR·DOUBLE
		 at
org.apache.sysml.lops.compile.Dag.generateControlProgramJobs
(Dag.java:1529)
		 at org.apache.sysml.lops.compile.Dag.doGreedyGrouping
(Dag.java:1194)
		 at org.apache.sysml.lops.compile.Dag.getJobs(Dag.java:268)
		 at
org.apache.sysml.parser.DMLProgram.createRuntimeProgramBlock
(DMLProgram.java:530)
		 at
org.apache.sysml.parser.DMLProgram.createRuntimeProgramBlock
(DMLProgram.java:327)
		 at org.apache.sysml.parser.DMLProgram.getRuntimeProgram
(DMLProgram.java:196)
		 at org.apache.sysml.api.DMLScript.execute(DMLScript.java:625)
		 at org.apache.sysml.api.DMLScript.executeScript
(DMLScript.java:337)
		 at org.apache.sysml.api.DMLScript.main(DMLScript.java:192)
		 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		 at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:62)
		 at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43)
		 at java.lang.reflect.Method.invoke(Method.java:498)
		 at org.apache.spark.deploy.SparkSubmit$.org$apache$spark
$deploy
$SparkSubmit$$runMain(SparkSubmit.scala:731)
		 at org.apache.spark.deploy.SparkSubmit$.doRunMain$1
(SparkSubmit.scala:181)
		 at org.apache.spark.deploy.SparkSubmit$.submit
(SparkSubmit.scala:206)
		 at org.apache.spark.deploy.SparkSubmit$.main
(SparkSubmit.scala:121)
		 at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NullPointerException
		 at
org.apache.sysml.runtime.instructions.SPInstructionParser.parseSingleInstruction

(SPInstructionParser.java:284)
		 at
org.apache.sysml.runtime.instructions.InstructionParser.parseSingleInstruction

(InstructionParser.java:47)
		 at
org.apache.sysml.lops.compile.Dag.generateControlProgramJobs
(Dag.java:1516)
		 ... 17 more
Exception in thread "main" org.apache.sysml.api.DMLException:
org.apache.sysml.lops.LopsException: ERROR: line 240, column 30 -- Problem
generating simple inst - SPARK°cdf°dist=normal°target=¶_Var237¶°
_Var238·SCALAR·DOUBLE
		 at org.apache.sysml.api.DMLScript.executeScript
(DMLScript.java:351)
		 at org.apache.sysml.api.DMLScript.main(DMLScript.java:192)
		 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		 at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:62)
		 at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43)
		 at java.lang.reflect.Method.invoke(Method.java:498)
		 at org.apache.spark.deploy.SparkSubmit$.org$apache$spark
$deploy
$SparkSubmit$$runMain(SparkSubmit.scala:731)
		 at org.apache.spark.deploy.SparkSubmit$.doRunMain$1
(SparkSubmit.scala:181)
		 at org.apache.spark.deploy.SparkSubmit$.submit
(SparkSubmit.scala:206)
		 at org.apache.spark.deploy.SparkSubmit$.main
(SparkSubmit.scala:121)
		 at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.sysml.lops.LopsException: ERROR: line 240, column 30
-- Problem generating simple inst - SPARK°cdf°dist=normal°target=¶_Var237¶°
_Var238·SCALAR·DOUBLE
		 at
org.apache.sysml.lops.compile.Dag.generateControlProgramJobs
(Dag.java:1529)
		 at org.apache.sysml.lops.compile.Dag.doGreedyGrouping
(Dag.java:1194)
		 at org.apache.sysml.lops.compile.Dag.getJobs(Dag.java:268)
		 at
org.apache.sysml.parser.DMLProgram.createRuntimeProgramBlock
(DMLProgram.java:530)
		 at
org.apache.sysml.parser.DMLProgram.createRuntimeProgramBlock
(DMLProgram.java:327)
		 at org.apache.sysml.parser.DMLProgram.getRuntimeProgram
(DMLProgram.java:196)
		 at org.apache.sysml.api.DMLScript.execute(DMLScript.java:625)
		 at org.apache.sysml.api.DMLScript.executeScript
(DMLScript.java:337)
		 ... 10 more
Caused by: java.lang.NullPointerException
		 at
org.apache.sysml.runtime.instructions.SPInstructionParser.parseSingleInstruction

(SPInstructionParser.java:284)
		 at
org.apache.sysml.runtime.instructions.InstructionParser.parseSingleInstruction

(InstructionParser.java:47)
		 at
org.apache.sysml.lops.compile.Dag.generateControlProgramJobs
(Dag.java:1516)
		 ... 17 more
16/09/02 15:35:10 INFO SparkContext: Invoking stop() from shutdown hook


Although I first hit the null pointer exception running newer code against
Spark 2.0, the stack trace above corresponds to 0.10.0 release against
Spark 1.6.1.  No problem occurs if run with -exec hybrid_spark.  I only
used -exec spark to force different scenario and just checking if anyone
else has seen this before opening a JIRA.

Thanks,
Glenn



Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message