systemml-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Glenn Weidner (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SYSTEMML-1071) Perftest: Univar-Stats.dml fails with dimension mismatch for M (8GB) scenario
Date Tue, 01 Nov 2016 20:32:58 GMT

    [ https://issues.apache.org/jira/browse/SYSTEMML-1071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15626597#comment-15626597
] 

Glenn Weidner commented on SYSTEMML-1071:
-----------------------------------------

To workaround test data generation issue [SYSTEMML-1044], used MR to successfully create 8GB
data:

    ./genDescriptiveStatisticsData_1M.sh gwperftest MR &>> logs/genStatsData.out

Univar-Stats.dml completed successfully (no dimension mismatch) against the complete data
set:

    ./runAllStats_8g_univar.sh gwperftest SPARK

SystemML execution statistics:

Total elapsed time:		747.936 sec.
Total compilation time:		2.032 sec.
Total execution time:		745.905 sec.
Number of compiled Spark inst:	0.
Number of executed Spark inst:	0.
Cache hits (Mem, WB, FS, HDFS):	27904/0/0/2.
Cache writes (WB, FS, HDFS):	3138/0/1.
Cache times (ACQr/m, RLS, EXP):	9.478/0.027/0.463/0.365 sec.
HOP DAGs recompiled (PRED, SB):	0/100.
HOP DAGs recompile time:	0.413 sec.
Spark ctx create time (lazy):	0.000 sec.
Spark trans counts (par,bc,col):0/0/0.
Spark trans times (par,bc,col):	0.000/0.000/0.000 secs.
ParFor loops optimized:		1.
ParFor optimize time:		0.179 sec.
ParFor initialize time:		0.010 sec.
ParFor result merge time:	0.002 sec.
ParFor total update in-place:	0/0/13900
Total JIT compile time:		4.29 sec.
Total JVM GC count:		25.
Total JVM GC time:		1.365 sec.
Heavy hitter instructions (name, time, count):
-- 1) 	cm 	463.780 sec 	2700
-- 2) 	qsort 	221.881 sec 	900
-- 3) 	rangeReIndex 	15.815 sec 	2999
-- 4) 	uamean 	12.531 sec 	900
-- 5) 	qpick 	12.011 sec 	1800
-- 6) 	uacmax 	10.099 sec 	1
-- 7) 	uamax 	2.754 sec 	1101
-- 8) 	uamin 	2.546 sec 	1000
-- 9) 	ctable 	2.444 sec 	100
-- 10) 	leftIndex 	0.432 sec 	13900


> Perftest: Univar-Stats.dml fails with dimension mismatch for M (8GB) scenario
> -----------------------------------------------------------------------------
>
>                 Key: SYSTEMML-1071
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1071
>             Project: SystemML
>          Issue Type: Bug
>            Reporter: Glenn Weidner
>
> Univar-Stats.dml fails for 8G medium scenario (runUnivar-Stats_A_1M) with below stack
trace.  Note smaller and larger (80G) scenarios completed without error.
> 16/10/27 10:20:20 WARN parser.Expression: WARNING: ../algorithms/Univar-Stats.dml --
line 64, column 21 -- ppred() has been deprecated. Please use the operator directly.
> 16/10/27 10:20:20 ERROR api.DMLScript: Failed to execute DML script.
> org.apache.sysml.parser.LanguageException: Invalid Parameters : ERROR: ../algorithms/Univar-Stats.dml
-- line 64, column 21 -- Mismatch in dimensions for operation (PPRED(K,1,>) MULT maxs)
> 	at org.apache.sysml.parser.Expression.raiseValidateError(Expression.java:557)
> 	at org.apache.sysml.parser.BinaryExpression.checkAndSetDimensions(BinaryExpression.java:188)
> 	at org.apache.sysml.parser.BinaryExpression.validateExpression(BinaryExpression.java:141)
> 	at org.apache.sysml.parser.BuiltinFunctionExpression.validateExpression(BuiltinFunctionExpression.java:323)
> 	at org.apache.sysml.parser.StatementBlock.validate(StatementBlock.java:600)
> 	at org.apache.sysml.parser.DMLTranslator.validateParseTree(DMLTranslator.java:136)
> 	at org.apache.sysml.api.DMLScript.execute(DMLScript.java:595)
> 	at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:354)
> 	at org.apache.sysml.api.DMLScript.main(DMLScript.java:199)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
> 	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
> 	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
> 	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
> 	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Exception in thread "main" org.apache.sysml.api.DMLException: org.apache.sysml.parser.LanguageException:
Invalid Parameters : ERROR: ../algorithms/Univar-Stats.dml -- line 64, column 21 -- Mismatch
in dimensions for operation (PPRED(K,1,>) MULT maxs)
> 	at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:368)
> 	at org.apache.sysml.api.DMLScript.main(DMLScript.java:199)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
> 	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
> 	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
> 	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
> 	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: org.apache.sysml.parser.LanguageException: Invalid Parameters : ERROR: ../algorithms/Univar-Stats.dml
-- line 64, column 21 -- Mismatch in dimensions for operation (PPRED(K,1,>) MULT maxs)
> 	at org.apache.sysml.parser.Expression.raiseValidateError(Expression.java:557)
> 	at org.apache.sysml.parser.BinaryExpression.checkAndSetDimensions(BinaryExpression.java:188)
> 	at org.apache.sysml.parser.BinaryExpression.validateExpression(BinaryExpression.java:141)
> 	at org.apache.sysml.parser.BuiltinFunctionExpression.validateExpression(BuiltinFunctionExpression.java:323)
> 	at org.apache.sysml.parser.StatementBlock.validate(StatementBlock.java:600)
> 	at org.apache.sysml.parser.DMLTranslator.validateParseTree(DMLTranslator.java:136)
> 	at org.apache.sysml.api.DMLScript.execute(DMLScript.java:595)
> 	at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:354)
> 	... 10 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message