systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ethan Xu <ethan.yifa...@gmail.com>
Subject Compatibility with MR1 Cloudera cdh4.2.1
Date Fri, 05 Feb 2016 01:51:16 GMT
Hello,

I got an error when running the systemML/scripts/Univar-Stats.dml script on
a hadoop cluster (Cloudera CDH4.2.1) on a 6GB data set. Error message is at
the bottom of the email. The same script ran fine on a smaller sample
(several MB) of the same data set, when MR was not invoked.

The main error was java.lang.NoSuchMethodError:
org.apache.hadoop.mapred.JobConf.getDouble()
Digging deeper, it looks like the CDH4.2.1 version of MR indeed didn't have
the JobConf.getDouble() method.

The hadoop-core jar of CDH4.2.1 can be found here:
https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-core/2.0.0-mr1-cdh4.2.1/

The calling line of SystemML is line 1194 of
https://github.com/apache/incubator-systemml/blob/master/src/main/java/org/apache/sysml/runtime/matrix/mapred/MRJobConfiguration.java

I was wondering, if the finding is accurate, is there a potential fix, or
does this mean the current version of SystemML is not compatible with
CDH4.2.1?

Thank you,

Ethan


hadoop jar $sysDir/target/SystemML.jar -f
$sysDir/scripts/algorithms/Univar-Stats.dml -nvargs
X=$baseDirHDFS/original-coded.csv
TYPES=$baseDirHDFS/original-coded-type.csv
STATS=$baseDirHDFS/univariate-summary.csv

16/02/04 20:35:03 INFO api.DMLScript: BEGIN DML run 02/04/2016 20:35:03
16/02/04 20:35:03 INFO api.DMLScript: HADOOP_HOME: null
16/02/04 20:35:03 WARN conf.DMLConfig: No default SystemML config file
(./SystemML-config.xml) found
16/02/04 20:35:03 WARN conf.DMLConfig: Using default settings in DMLConfig
16/02/04 20:35:04 WARN hops.OptimizerUtils: Auto-disable multi-threaded
text read for 'text' and 'csv' due to thread contention on JRE < 1.8
(java.version=1.7.0_71).
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/explorys/datagrid/lib/slf4j-jdk14-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/explorys/datagrid/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
16/02/04 20:35:07 INFO api.DMLScript: SystemML Statistics:
Total execution time:        0.880 sec.
Number of executed MR Jobs:    0.

16/02/04 20:35:07 INFO api.DMLScript: END DML run 02/04/2016 20:35:07
Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.hadoop.mapred.JobConf.getDouble(Ljava/lang/String;D)D
    at
org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1195)
    at
org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1129)
    at
org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:307)
    at
org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:289)
    at
org.apache.sysml.runtime.matrix.CSVReblockMR.runJob(CSVReblockMR.java:275)
    at org.apache.sysml.lops.runtime.RunMRJobs.submitJob(RunMRJobs.java:257)
    at
org.apache.sysml.lops.runtime.RunMRJobs.prepareAndSubmitJob(RunMRJobs.java:143)
    at
org.apache.sysml.runtime.instructions.MRJobInstruction.processInstruction(MRJobInstruction.java:1500)
    at
org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:309)
    at
org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:227)
    at
org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:169)
    at
org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:146)
    at org.apache.sysml.api.DMLScript.execute(DMLScript.java:676)
    at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:338)
    at org.apache.sysml.api.DMLScript.main(DMLScript.java:197)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message