systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias Boehm" <mbo...@us.ibm.com>
Subject Re: Compatibility with MR1 Cloudera cdh4.2.1
Date Fri, 05 Feb 2016 02:40:29 GMT

well, we did indeed not run on MR v1 for a while now. However, I don't want
to get that far and say we don't support it anymore. I'll fix this
particular issue by tomorrow.

In the next couple of weeks we should run our full performance testsuite
(for broad coverage) over an MR v1 cluster and systematically remove
unnecessary incompatibility like this instance. Any volunteers?

Regards,
Matthias



From:	Ethan Xu <ethan.yifanxu@gmail.com>
To:	dev@systemml.incubator.apache.org
Date:	02/04/2016 05:51 PM
Subject:	Compatibility with MR1 Cloudera cdh4.2.1



Hello,

I got an error when running the systemML/scripts/Univar-Stats.dml script on
a hadoop cluster (Cloudera CDH4.2.1) on a 6GB data set. Error message is at
the bottom of the email. The same script ran fine on a smaller sample
(several MB) of the same data set, when MR was not invoked.

The main error was java.lang.NoSuchMethodError:
org.apache.hadoop.mapred.JobConf.getDouble()
Digging deeper, it looks like the CDH4.2.1 version of MR indeed didn't have
the JobConf.getDouble() method.

The hadoop-core jar of CDH4.2.1 can be found here:
https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-core/2.0.0-mr1-cdh4.2.1/


The calling line of SystemML is line 1194 of
https://github.com/apache/incubator-systemml/blob/master/src/main/java/org/apache/sysml/runtime/matrix/mapred/MRJobConfiguration.java


I was wondering, if the finding is accurate, is there a potential fix, or
does this mean the current version of SystemML is not compatible with
CDH4.2.1?

Thank you,

Ethan


hadoop jar $sysDir/target/SystemML.jar -f
$sysDir/scripts/algorithms/Univar-Stats.dml -nvargs
X=$baseDirHDFS/original-coded.csv
TYPES=$baseDirHDFS/original-coded-type.csv
STATS=$baseDirHDFS/univariate-summary.csv

16/02/04 20:35:03 INFO api.DMLScript: BEGIN DML run 02/04/2016 20:35:03
16/02/04 20:35:03 INFO api.DMLScript: HADOOP_HOME: null
16/02/04 20:35:03 WARN conf.DMLConfig: No default SystemML config file
(./SystemML-config.xml) found
16/02/04 20:35:03 WARN conf.DMLConfig: Using default settings in DMLConfig
16/02/04 20:35:04 WARN hops.OptimizerUtils: Auto-disable multi-threaded
text read for 'text' and 'csv' due to thread contention on JRE < 1.8
(java.version=1.7.0_71).
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in
[jar:file:/usr/local/explorys/datagrid/lib/slf4j-jdk14-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in
[jar:file:/usr/local/explorys/datagrid/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
16/02/04 20:35:07 INFO api.DMLScript: SystemML Statistics:
Total execution time:        0.880 sec.
Number of executed MR Jobs:    0.

16/02/04 20:35:07 INFO api.DMLScript: END DML run 02/04/2016 20:35:07
Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.hadoop.mapred.JobConf.getDouble(Ljava/lang/String;D)D
    at
org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs
(MRJobConfiguration.java:1195)
    at
org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs
(MRJobConfiguration.java:1129)
    at
org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob
(CSVReblockMR.java:307)
    at
org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob
(CSVReblockMR.java:289)
    at
org.apache.sysml.runtime.matrix.CSVReblockMR.runJob(CSVReblockMR.java:275)
    at org.apache.sysml.lops.runtime.RunMRJobs.submitJob
(RunMRJobs.java:257)
    at
org.apache.sysml.lops.runtime.RunMRJobs.prepareAndSubmitJob
(RunMRJobs.java:143)
    at
org.apache.sysml.runtime.instructions.MRJobInstruction.processInstruction
(MRJobInstruction.java:1500)
    at
org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction
(ProgramBlock.java:309)
    at
org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions
(ProgramBlock.java:227)
    at
org.apache.sysml.runtime.controlprogram.ProgramBlock.execute
(ProgramBlock.java:169)
    at
org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:146)
    at org.apache.sysml.api.DMLScript.execute(DMLScript.java:676)
    at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:338)
    at org.apache.sysml.api.DMLScript.main(DMLScript.java:197)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:57)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)


Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message