systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Glenn Weidner" <gweid...@us.ibm.com>
Subject NPE running GLM-predict.dml in spark mode
Date Fri, 02 Sep 2016 23:00:09 GMT


When running Linear Regression example described at
https://apache.github.io/incubator-systemml/standalone-guide.html#linear-regression-example
, everything works well with spark-submit under hybrid_spark mode.
However, when running the very last step GLM-predict with -exec spark, a
null pointer exception occurs.

spark-submit --master yarn --deploy-mode client --class
org.apache.sysml.api.DMLScript systemml.jar -f GLM-predict.dml -exec spark
-nvargs X=linRegData.test.data.csv Y=linRegData.test.labels.csv B=betas.csv
fmt=csv

16/09/02 15:35:10 ERROR DMLScript: Failed to execute DML script.
org.apache.sysml.lops.LopsException: ERROR: line 240, column 30 -- Problem
generating simple inst - SPARK°cdf°dist=normal°target=¶_Var237¶°
_Var238·SCALAR·DOUBLE
	at org.apache.sysml.lops.compile.Dag.generateControlProgramJobs
(Dag.java:1529)
	at org.apache.sysml.lops.compile.Dag.doGreedyGrouping(Dag.java:1194)
	at org.apache.sysml.lops.compile.Dag.getJobs(Dag.java:268)
	at org.apache.sysml.parser.DMLProgram.createRuntimeProgramBlock
(DMLProgram.java:530)
	at org.apache.sysml.parser.DMLProgram.createRuntimeProgramBlock
(DMLProgram.java:327)
	at org.apache.sysml.parser.DMLProgram.getRuntimeProgram
(DMLProgram.java:196)
	at org.apache.sysml.api.DMLScript.execute(DMLScript.java:625)
	at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:337)
	at org.apache.sysml.api.DMLScript.main(DMLScript.java:192)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy
$SparkSubmit$$runMain(SparkSubmit.scala:731)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1
(SparkSubmit.scala:181)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NullPointerException
	at
org.apache.sysml.runtime.instructions.SPInstructionParser.parseSingleInstruction
(SPInstructionParser.java:284)
	at
org.apache.sysml.runtime.instructions.InstructionParser.parseSingleInstruction
(InstructionParser.java:47)
	at org.apache.sysml.lops.compile.Dag.generateControlProgramJobs
(Dag.java:1516)
	... 17 more
Exception in thread "main" org.apache.sysml.api.DMLException:
org.apache.sysml.lops.LopsException: ERROR: line 240, column 30 -- Problem
generating simple inst - SPARK°cdf°dist=normal°target=¶_Var237¶°
_Var238·SCALAR·DOUBLE
	at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:351)
	at org.apache.sysml.api.DMLScript.main(DMLScript.java:192)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy
$SparkSubmit$$runMain(SparkSubmit.scala:731)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1
(SparkSubmit.scala:181)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.sysml.lops.LopsException: ERROR: line 240, column 30
-- Problem generating simple inst - SPARK°cdf°dist=normal°target=¶_Var237¶°
_Var238·SCALAR·DOUBLE
	at org.apache.sysml.lops.compile.Dag.generateControlProgramJobs
(Dag.java:1529)
	at org.apache.sysml.lops.compile.Dag.doGreedyGrouping(Dag.java:1194)
	at org.apache.sysml.lops.compile.Dag.getJobs(Dag.java:268)
	at org.apache.sysml.parser.DMLProgram.createRuntimeProgramBlock
(DMLProgram.java:530)
	at org.apache.sysml.parser.DMLProgram.createRuntimeProgramBlock
(DMLProgram.java:327)
	at org.apache.sysml.parser.DMLProgram.getRuntimeProgram
(DMLProgram.java:196)
	at org.apache.sysml.api.DMLScript.execute(DMLScript.java:625)
	at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:337)
	... 10 more
Caused by: java.lang.NullPointerException
	at
org.apache.sysml.runtime.instructions.SPInstructionParser.parseSingleInstruction
(SPInstructionParser.java:284)
	at
org.apache.sysml.runtime.instructions.InstructionParser.parseSingleInstruction
(InstructionParser.java:47)
	at org.apache.sysml.lops.compile.Dag.generateControlProgramJobs
(Dag.java:1516)
	... 17 more
16/09/02 15:35:10 INFO SparkContext: Invoking stop() from shutdown hook


Although I first hit the null pointer exception running newer code against
Spark 2.0, the stack trace above corresponds to 0.10.0 release against
Spark 1.6.1.  No problem occurs if run with -exec hybrid_spark.  I only
used -exec spark to force different scenario and just checking if anyone
else has seen this before opening a JIRA.

Thanks,
Glenn

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message