systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Glenn Weidner" <gweid...@us.ibm.com>
Subject Re: Running the bivar-stats example
Date Thu, 27 Oct 2016 20:22:13 GMT
Hi Sandeep,

No problem and thank you for trying the other scenarios.  Just one quick
question - are you using the 0.10.0 binary release?

Thanks,
Glenn




From:	"Narayanaswami, Sandeep" <Sandeep.Narayanaswami@capitalone.com>
To:	"dev@systemml.incubator.apache.org"
            <dev@systemml.incubator.apache.org>
Date:	10/27/2016 01:14 PM
Subject:	Re: Running the bivar-stats example



Hi, sorry I couldn’t get back on this sooner.

@Glenn: Thanks for pointing that out. Unfortunately, switching to Spark 1.6
didn’t make any difference: I got the same error/stacktrace again.
@Matthias: I did have the corresponding metadata files. I also tried
modifying the script to print the matrix after reading it in, and that
works as expected so I don’t think it’s a data read issue.

I’d appreciate any other suggestions.

Cheers,
Sandeep

On 10/20/16, 02:30, "Matthias Boehm" <mboehm7@googlemail.com> wrote:

    apart from the missing support for Spark 2.x, I would recommend to
    double check your inputs. Do you have meta data files along with your
    inputs? If not, then these inputs are assumed to be in default format
    "text", i.e., matrix market ijv representation (not csv). You can
either
    provide such json meta data files or simply put a 'format="csv"' into
    the read statements in bivar-stats.dml.

    Regards,
    Matthias


    On 10/20/2016 2:44 AM, Glenn Weidner wrote:
    > Hi Sandeep,
    >
    > Thank you for contacting the Apache SystemML community.  Based on the
    > following trace fragment:
    >
    > spark.SparkContext: Running Spark version 2.0.0
    >
    > it looks like you're running against Spark 2.0 which is not
officially
    > supported with the current master code base.  Can you retry with an
earlier
    > Spark version?  Let me know if I've misinterpreted your Spark
environment.
    >
    > Cheers,
    > Glenn
    >
    >
    >
    > From:		 "Narayanaswami, Sandeep"
<Sandeep.Narayanaswami@capitalone.com>
    > To:		 "dev@systemml.incubator.apache.org"
    >             <dev@systemml.incubator.apache.org>
    > Date:		 10/19/2016 05:00 PM
    > Subject:		 Running the bivar-stats example
    >
    >
    >
    > Hi SystemML community,
    >
    > I’m trying to run the bivariate stats example script on Spark. I’m
only
    > doing a toy example with very small data, but I have been running
into
    > errors.
    > My script invocation is as follows:
    >
    > spark-submit --master local[4] SystemML.jar -f
    > scripts/algorithms/bivar-stats.dml -exec hybrid_spark -nvargs
    > X=/data/iqtv.csv index1=/data/index1.csv index2=/data/index2.csv
    > types1=/data/types1.csv types2=/data/types2.csv OUTDIR=/data/outdata
    >
    > Am I calling the script incorrectly?
    > (All the files are on hdfs. Iqtv.csv is a headerless csv with two
numeric
    > fields. The index1 and index2 files are text files containing only
the
    > strings ‘1’ and ‘2’ respectively. The types* files are text files
each
    > containing only the string ‘1’.)
    >
    > Please excuse the rather long stack trace below, but I’m hoping it
will
    > help diagnose the problem.
    >
    > Cheers,
    > Sandeep
    >
    >
================================================================================================================================

    >
    >
    > 16/10/19 16:46:26 INFO api.DMLScript: BEGIN DML run 10/19/2016
16:46:26
    > ANTLR Tool version 4.3 used for code generation does not match the
current
    > runtime version 4.5.3[…]
    > 16/10/19 16:46:28 INFO spark.SparkContext: Running Spark version
2.0.0
    > […]
    > 16/10/19 16:46:29 INFO api.DMLScript: END DML run 10/19/2016 16:46:29
    > 16/10/19 16:46:29 ERROR api.DMLScript: Failed to execute DML script.
    > org.apache.sysml.runtime.DMLRuntimeException:
    > org.apache.sysml.runtime.DMLRuntimeException:
    > org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile
program
    > block hierarchy.
    >        at org.apache.sysml.runtime.controlprogram.Program.execute
    > (Program.java:152)
    >        at org.apache.sysml.api.DMLScript.execute(DMLScript.java:677)
    >        at org.apache.sysml.api.DMLScript.executeScript
(DMLScript.java:337)
    >        at org.apache.sysml.api.DMLScript.main(DMLScript.java:192)
    >        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    >        at sun.reflect.NativeMethodAccessorImpl.invoke
    > (NativeMethodAccessorImpl.java:62)
    >        at sun.reflect.DelegatingMethodAccessorImpl.invoke
    > (DelegatingMethodAccessorImpl.java:43)
    >        at java.lang.reflect.Method.invoke(Method.java:497)
    >        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark
$deploy
    > $SparkSubmit$$runMain(SparkSubmit.scala:729)
    >        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1
    > (SparkSubmit.scala:185)
    >        at org.apache.spark.deploy.SparkSubmit$.submit
    > (SparkSubmit.scala:210)
    >        at org.apache.spark.deploy.SparkSubmit$.main
(SparkSubmit.scala:124)
    >        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    > Caused by: org.apache.sysml.runtime.DMLRuntimeException:
    > org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile
program
    > block hierarchy.
    >        at
    >
org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize

    > (OptimizationWrapper.java:299)
    >        at
    >
org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize

    > (OptimizationWrapper.java:172)
    >        at
    > org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute
    > (ParForProgramBlock.java:550)
    >        at org.apache.sysml.runtime.controlprogram.Program.execute
    > (Program.java:145)
    >        ... 12 more
    > Caused by: org.apache.sysml.runtime.DMLRuntimeException: Unable to
    > recompile program block hierarchy.
    >        at
    >
org.apache.sysml.hops.recompile.Recompiler.recompileProgramBlockHierarchy
    > (Recompiler.java:541)
    >        at
    >
org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize

    > (OptimizationWrapper.java:278)
    >        ... 15 more
    > Caused by: org.apache.sysml.hops.HopsException: ERROR: line 342,
column 4
    > -- error constructing Lops for UnaryOp Hop --
    >
    >        at org.apache.sysml.hops.UnaryOp.constructLops
(UnaryOp.java:177)
    >        at org.apache.sysml.hops.DataOp.constructLops(DataOp.java:252)
    >        at org.apache.sysml.hops.recompile.Recompiler.recompileHopsDag
    > (Recompiler.java:209)
    >        at
org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock
    > (Recompiler.java:881)
    >        at
org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock
    > (Recompiler.java:839)
    >        at
org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock
    > (Recompiler.java:839)
    >        at
org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock
    > (Recompiler.java:841)
    >        at
org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock
    > (Recompiler.java:858)
    >        at
    >
org.apache.sysml.hops.recompile.Recompiler.recompileProgramBlockHierarchy
    > (Recompiler.java:536)
    >        ... 16 more
    > Caused by: org.apache.sysml.hops.HopsException: ERROR: line 0, column
0 --
    > error constructing Lops for UnaryOp Hop --
    >
    >        at org.apache.sysml.hops.UnaryOp.constructLops
(UnaryOp.java:177)
    >        at org.apache.sysml.hops.BinaryOp.constructLopsBinaryDefault
    > (BinaryOp.java:565)
    >        at org.apache.sysml.hops.BinaryOp.constructLops
(BinaryOp.java:166)
    >        at org.apache.sysml.hops.BinaryOp.constructLopsBinaryDefault
    > (BinaryOp.java:566)
    >        at org.apache.sysml.hops.BinaryOp.constructLops
(BinaryOp.java:166)
    >        at org.apache.sysml.hops.UnaryOp.constructLops
(UnaryOp.java:142)
    >        ... 24 more
    > Caused by: java.lang.NullPointerException
    >        at
    >
org.apache.sysml.hops.ParameterizedBuiltinOp.constructLopsGroupedAggregate
    > (ParameterizedBuiltinOp.java:247)
    >        at org.apache.sysml.hops.ParameterizedBuiltinOp.constructLops
    > (ParameterizedBuiltinOp.java:181)
    >        at org.apache.sysml.hops.BinaryOp.constructLopsBinaryDefault
    > (BinaryOp.java:590)
    >        at org.apache.sysml.hops.BinaryOp.constructLops
(BinaryOp.java:166)
    >        at org.apache.sysml.hops.ReorgOp.constructLops
(ReorgOp.java:128)
    >        at org.apache.sysml.hops.AggBinaryOp.constructSparkLopsCPMM
    > (AggBinaryOp.java:832)
    >        at org.apache.sysml.hops.AggBinaryOp.constructLops
    > (AggBinaryOp.java:216)
    >        at org.apache.sysml.hops.UnaryOp.constructLops
(UnaryOp.java:142)
    >        ... 29 more
    > Exception in thread "main" org.apache.sysml.api.DMLException:
    > org.apache.sysml.runtime.DMLRuntimeException:
    > org.apache.sysml.runtime.DMLRuntimeException:
    > org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile
program
    > block hierarchy.
    >        at org.apache.sysml.api.DMLScript.executeScript
(DMLScript.java:351)
    >        at org.apache.sysml.api.DMLScript.main(DMLScript.java:192)
    >        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    >        at sun.reflect.NativeMethodAccessorImpl.invoke
    > (NativeMethodAccessorImpl.java:62)
    >        at sun.reflect.DelegatingMethodAccessorImpl.invoke
    > (DelegatingMethodAccessorImpl.java:43)
    >        at java.lang.reflect.Method.invoke(Method.java:497)
    >        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark
$deploy
    > $SparkSubmit$$runMain(SparkSubmit.scala:729)
    >        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1
    > (SparkSubmit.scala:185)
    >        at org.apache.spark.deploy.SparkSubmit$.submit
    > (SparkSubmit.scala:210)
    >        at org.apache.spark.deploy.SparkSubmit$.main
(SparkSubmit.scala:124)
    >        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    > Caused by: org.apache.sysml.runtime.DMLRuntimeException:
    > org.apache.sysml.runtime.DMLRuntimeException:
    > org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile
program
    > block hierarchy.
    >        at org.apache.sysml.runtime.controlprogram.Program.execute
    > (Program.java:152)
    >        at org.apache.sysml.api.DMLScript.execute(DMLScript.java:677)
    >        at org.apache.sysml.api.DMLScript.executeScript
(DMLScript.java:337)
    >        ... 10 more
    > Caused by: org.apache.sysml.runtime.DMLRuntimeException:
    > org.apache.sysml.runtime.DMLRuntimeException: Unable to recompile
program
    > block hierarchy.
    >        at
    >
org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize

    > (OptimizationWrapper.java:299)
    >        at
    >
org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize

    > (OptimizationWrapper.java:172)
    >        at
    > org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute
    > (ParForProgramBlock.java:550)
    >        at org.apache.sysml.runtime.controlprogram.Program.execute
    > (Program.java:145)
    >        ... 12 more
    > Caused by: org.apache.sysml.runtime.DMLRuntimeException: Unable to
    > recompile program block hierarchy.
    >        at
    >
org.apache.sysml.hops.recompile.Recompiler.recompileProgramBlockHierarchy
    > (Recompiler.java:541)
    >        at
    >
org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize

    > (OptimizationWrapper.java:278)
    >        ... 15 more
    > Caused by: org.apache.sysml.hops.HopsException: ERROR: line 342,
column 4
    > -- error constructing Lops for UnaryOp Hop --
    >
    >        at org.apache.sysml.hops.UnaryOp.constructLops
(UnaryOp.java:177)
    >        at org.apache.sysml.hops.DataOp.constructLops(DataOp.java:252)
    >        at org.apache.sysml.hops.recompile.Recompiler.recompileHopsDag
    > (Recompiler.java:209)
    >        at
org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock
    > (Recompiler.java:881)
    >        at
org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock
    > (Recompiler.java:839)
    >        at
org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock
    > (Recompiler.java:839)
    >        at
org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock
    > (Recompiler.java:841)
    >        at
org.apache.sysml.hops.recompile.Recompiler.rRecompileProgramBlock
    > (Recompiler.java:858)
    >        at
    >
org.apache.sysml.hops.recompile.Recompiler.recompileProgramBlockHierarchy
    > (Recompiler.java:536)
    >        ... 16 more
    > Caused by: org.apache.sysml.hops.HopsException: ERROR: line 0, column
0 --
    > error constructing Lops for UnaryOp Hop --
    >
    >        at org.apache.sysml.hops.UnaryOp.constructLops
(UnaryOp.java:177)
    >        at org.apache.sysml.hops.BinaryOp.constructLopsBinaryDefault
    > (BinaryOp.java:565)
    >        at org.apache.sysml.hops.BinaryOp.constructLops
(BinaryOp.java:166)
    >        at org.apache.sysml.hops.BinaryOp.constructLopsBinaryDefault
    > (BinaryOp.java:566)
    >        at org.apache.sysml.hops.BinaryOp.constructLops
(BinaryOp.java:166)
    >        at org.apache.sysml.hops.UnaryOp.constructLops
(UnaryOp.java:142)
    >        ... 24 more
    > Caused by: java.lang.NullPointerException
    >        at
    >
org.apache.sysml.hops.ParameterizedBuiltinOp.constructLopsGroupedAggregate
    > (ParameterizedBuiltinOp.java:247)
    >        at org.apache.sysml.hops.ParameterizedBuiltinOp.constructLops
    > (ParameterizedBuiltinOp.java:181)
    >        at org.apache.sysml.hops.BinaryOp.constructLopsBinaryDefault
    > (BinaryOp.java:590)
    >        at org.apache.sysml.hops.BinaryOp.constructLops
(BinaryOp.java:166)
    >        at org.apache.sysml.hops.ReorgOp.constructLops
(ReorgOp.java:128)
    >        at org.apache.sysml.hops.AggBinaryOp.constructSparkLopsCPMM
    > (AggBinaryOp.java:832)
    >        at org.apache.sysml.hops.AggBinaryOp.constructLops
    > (AggBinaryOp.java:216)
    >        at org.apache.sysml.hops.UnaryOp.constructLops
(UnaryOp.java:142)
    >        ... 29 more
    > 16/10/19 16:46:29 INFO util.ShutdownHookManager: Shutdown hook called
    > […]
    >
    > ________________________________________________________
    >
    > The information contained in this e-mail is confidential and/or
proprietary
    > to Capital One and/or its affiliates and may only be used solely in
    > performance of work or services for Capital One. The information
    > transmitted herewith is intended only for use by the individual or
entity
    > to which it is addressed. If the reader of this message is not the
intended
    > recipient, you are hereby notified that any review, retransmission,
    > dissemination, distribution, copying or other use of, or taking of
any
    > action in reliance upon this information is strictly prohibited. If
you
    > have received this communication in error, please contact the sender
and
    > delete the material from your computer.
    >
    >




________________________________________________________

The information contained in this e-mail is confidential and/or proprietary
to Capital One and/or its affiliates and may only be used solely in
performance of work or services for Capital One. The information
transmitted herewith is intended only for use by the individual or entity
to which it is addressed. If the reader of this message is not the intended
recipient, you are hereby notified that any review, retransmission,
dissemination, distribution, copying or other use of, or taking of any
action in reliance upon this information is strictly prohibited. If you
have received this communication in error, please contact the sender and
delete the material from your computer.


Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message