systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Narayanaswami, Sandeep" <Sandeep.Narayanasw...@capitalone.com>
Subject Running the bivar-stats example
Date Wed, 19 Oct 2016 23:59:48 GMT
Hi SystemML community,

I’m trying to run the bivariate stats example script on Spark. I’m only doing a toy example
with very small data, but I have been running into errors.
My script invocation is as follows:

spark-submit --master local[4] SystemML.jar -f scripts/algorithms/bivar-stats.dml -exec hybrid_spark
-nvargs X=/data/iqtv.csv index1=/data/index1.csv index2=/data/index2.csv types1=/data/types1.csv
types2=/data/types2.csv OUTDIR=/data/outdata

Am I calling the script incorrectly?
(All the files are on hdfs. Iqtv.csv is a headerless csv with two numeric fields. The index1
and index2 files are text files containing only the strings ‘1’ and ‘2’ respectively.
The types* files are text files each containing only the string ‘1’.)

Please excuse the rather long stack trace below, but I’m hoping it will help diagnose the
problem.

Cheers,
Sandeep

================================================================================================================================

16/10/19 16:46:26 INFO api.DMLScript: BEGIN DML run 10/19/2016 16:46:26
ANTLR Tool version 4.3 used for code generation does not match the current runtime version
4.5.3[…]
16/10/19 16:46:28 INFO spark.SparkContext: Running Spark version 2.0.0
[…]
16/10/19 16:46:29 INFO api.DMLScript: END DML run 10/19/2016 16:46:29
16/10/19 16:46:29 ERROR api.DMLScript: Failed to execute DML script.
org.apache.sysml.runtime.DMLRuntimeException: org.apache.sysml.runtime.DMLRuntimeException:
org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile program block hierarchy.
       at org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:152)
       at org.apache.sysml.api.DMLScript.execute(DMLScript.java:677)
       at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:337)
       at org.apache.sysml.api.DMLScript.main(DMLScript.java:192)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:497)
       at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729)
       at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
       at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
       at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
       at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.sysml.runtime.DMLRuntimeException: org.apache.sysml.runtime.DMLRuntimeException:
Unable to recompile program block hierarchy.
       at org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize(OptimizationWrapper.java:299)
       at org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize(OptimizationWrapper.java:172)
       at org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:550)
       at org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:145)
       ... 12 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile program block
hierarchy.
       at org.apache.sysml.hops.recompile.Recompiler.recompileProgramBlockHierarchy(Recompiler.java:541)
       at org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize(OptimizationWrapper.java:278)
       ... 15 more
Caused by: org.apache.sysml.hops.HopsException: ERROR: line 342, column 4 -- error constructing
Lops for UnaryOp Hop --

       at org.apache.sysml.hops.UnaryOp.constructLops(UnaryOp.java:177)
       at org.apache.sysml.hops.DataOp.constructLops(DataOp.java:252)
       at org.apache.sysml.hops.recompile.Recompiler.recompileHopsDag(Recompiler.java:209)
       at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock(Recompiler.java:881)
       at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock(Recompiler.java:839)
       at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock(Recompiler.java:839)
       at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock(Recompiler.java:841)
       at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock(Recompiler.java:858)
       at org.apache.sysml.hops.recompile.Recompiler.recompileProgramBlockHierarchy(Recompiler.java:536)
       ... 16 more
Caused by: org.apache.sysml.hops.HopsException: ERROR: line 0, column 0 -- error constructing
Lops for UnaryOp Hop --

       at org.apache.sysml.hops.UnaryOp.constructLops(UnaryOp.java:177)
       at org.apache.sysml.hops.BinaryOp.constructLopsBinaryDefault(BinaryOp.java:565)
       at org.apache.sysml.hops.BinaryOp.constructLops(BinaryOp.java:166)
       at org.apache.sysml.hops.BinaryOp.constructLopsBinaryDefault(BinaryOp.java:566)
       at org.apache.sysml.hops.BinaryOp.constructLops(BinaryOp.java:166)
       at org.apache.sysml.hops.UnaryOp.constructLops(UnaryOp.java:142)
       ... 24 more
Caused by: java.lang.NullPointerException
       at org.apache.sysml.hops.ParameterizedBuiltinOp.constructLopsGroupedAggregate(ParameterizedBuiltinOp.java:247)
       at org.apache.sysml.hops.ParameterizedBuiltinOp.constructLops(ParameterizedBuiltinOp.java:181)
       at org.apache.sysml.hops.BinaryOp.constructLopsBinaryDefault(BinaryOp.java:590)
       at org.apache.sysml.hops.BinaryOp.constructLops(BinaryOp.java:166)
       at org.apache.sysml.hops.ReorgOp.constructLops(ReorgOp.java:128)
       at org.apache.sysml.hops.AggBinaryOp.constructSparkLopsCPMM(AggBinaryOp.java:832)
       at org.apache.sysml.hops.AggBinaryOp.constructLops(AggBinaryOp.java:216)
       at org.apache.sysml.hops.UnaryOp.constructLops(UnaryOp.java:142)
       ... 29 more
Exception in thread "main" org.apache.sysml.api.DMLException: org.apache.sysml.runtime.DMLRuntimeException:
org.apache.sysml.runtime.DMLRuntimeException: org.apache.sysml.runtime.DMLRuntimeException:
Unable to recompile program block hierarchy.
       at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:351)
       at org.apache.sysml.api.DMLScript.main(DMLScript.java:192)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:497)
       at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729)
       at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
       at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
       at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
       at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.sysml.runtime.DMLRuntimeException: org.apache.sysml.runtime.DMLRuntimeException:
org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile program block hierarchy.
       at org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:152)
       at org.apache.sysml.api.DMLScript.execute(DMLScript.java:677)
       at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:337)
       ... 10 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: org.apache.sysml.runtime.DMLRuntimeException:
Unable to recompile program block hierarchy.
       at org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize(OptimizationWrapper.java:299)
       at org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize(OptimizationWrapper.java:172)
       at org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:550)
       at org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:145)
       ... 12 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile program block
hierarchy.
       at org.apache.sysml.hops.recompile.Recompiler.recompileProgramBlockHierarchy(Recompiler.java:541)
       at org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize(OptimizationWrapper.java:278)
       ... 15 more
Caused by: org.apache.sysml.hops.HopsException: ERROR: line 342, column 4 -- error constructing
Lops for UnaryOp Hop --

       at org.apache.sysml.hops.UnaryOp.constructLops(UnaryOp.java:177)
       at org.apache.sysml.hops.DataOp.constructLops(DataOp.java:252)
       at org.apache.sysml.hops.recompile.Recompiler.recompileHopsDag(Recompiler.java:209)
       at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock(Recompiler.java:881)
       at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock(Recompiler.java:839)
       at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock(Recompiler.java:839)
       at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock(Recompiler.java:841)
       at org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock(Recompiler.java:858)
       at org.apache.sysml.hops.recompile.Recompiler.recompileProgramBlockHierarchy(Recompiler.java:536)
       ... 16 more
Caused by: org.apache.sysml.hops.HopsException: ERROR: line 0, column 0 -- error constructing
Lops for UnaryOp Hop --

       at org.apache.sysml.hops.UnaryOp.constructLops(UnaryOp.java:177)
       at org.apache.sysml.hops.BinaryOp.constructLopsBinaryDefault(BinaryOp.java:565)
       at org.apache.sysml.hops.BinaryOp.constructLops(BinaryOp.java:166)
       at org.apache.sysml.hops.BinaryOp.constructLopsBinaryDefault(BinaryOp.java:566)
       at org.apache.sysml.hops.BinaryOp.constructLops(BinaryOp.java:166)
       at org.apache.sysml.hops.UnaryOp.constructLops(UnaryOp.java:142)
       ... 24 more
Caused by: java.lang.NullPointerException
       at org.apache.sysml.hops.ParameterizedBuiltinOp.constructLopsGroupedAggregate(ParameterizedBuiltinOp.java:247)
       at org.apache.sysml.hops.ParameterizedBuiltinOp.constructLops(ParameterizedBuiltinOp.java:181)
       at org.apache.sysml.hops.BinaryOp.constructLopsBinaryDefault(BinaryOp.java:590)
       at org.apache.sysml.hops.BinaryOp.constructLops(BinaryOp.java:166)
       at org.apache.sysml.hops.ReorgOp.constructLops(ReorgOp.java:128)
       at org.apache.sysml.hops.AggBinaryOp.constructSparkLopsCPMM(AggBinaryOp.java:832)
       at org.apache.sysml.hops.AggBinaryOp.constructLops(AggBinaryOp.java:216)
       at org.apache.sysml.hops.UnaryOp.constructLops(UnaryOp.java:142)
       ... 29 more
16/10/19 16:46:29 INFO util.ShutdownHookManager: Shutdown hook called
[…]

________________________________________________________

The information contained in this e-mail is confidential and/or proprietary to Capital One
and/or its affiliates and may only be used solely in performance of work or services for Capital
One. The information transmitted herewith is intended only for use by the individual or entity
to which it is addressed. If the reader of this message is not the intended recipient, you
are hereby notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is strictly prohibited.
If you have received this communication in error, please contact the sender and delete the
material from your computer.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message