systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shirish Tatikonda <shirish.tatiko...@gmail.com>
Subject Re: Compatibility with MR1 Cloudera cdh4.2.1
Date Fri, 05 Feb 2016 02:39:49 GMT
Hi Ethan,

The getDouble() method is actually part of
org.apache.hadoop.conf.Configuration.java, which is part of hadoop-common
but not hadoop-core -- see [1]. Seems like, it used to be part of
hadoop-core a long time ago.
Also, the pom.xml in SystemML project does specify hadoop-common as the
dependency (as provided library). Make sure hadoop-common jar file is in
the classpath.

[1]
http://grepcode.com/file/repo1.maven.org/maven2/com.ning/metrics.collector/1.3.0/org/apache/hadoop/conf/Configuration.java

Shirish


On Thu, Feb 4, 2016 at 5:51 PM, Ethan Xu <ethan.yifanxu@gmail.com> wrote:

> Hello,
>
> I got an error when running the systemML/scripts/Univar-Stats.dml script on
> a hadoop cluster (Cloudera CDH4.2.1) on a 6GB data set. Error message is at
> the bottom of the email. The same script ran fine on a smaller sample
> (several MB) of the same data set, when MR was not invoked.
>
> The main error was java.lang.NoSuchMethodError:
> org.apache.hadoop.mapred.JobConf.getDouble()
> Digging deeper, it looks like the CDH4.2.1 version of MR indeed didn't have
> the JobConf.getDouble() method.
>
> The hadoop-core jar of CDH4.2.1 can be found here:
>
> https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-core/2.0.0-mr1-cdh4.2.1/
>
> The calling line of SystemML is line 1194 of
>
> https://github.com/apache/incubator-systemml/blob/master/src/main/java/org/apache/sysml/runtime/matrix/mapred/MRJobConfiguration.java
>
> I was wondering, if the finding is accurate, is there a potential fix, or
> does this mean the current version of SystemML is not compatible with
> CDH4.2.1?
>
> Thank you,
>
> Ethan
>
>
> hadoop jar $sysDir/target/SystemML.jar -f
> $sysDir/scripts/algorithms/Univar-Stats.dml -nvargs
> X=$baseDirHDFS/original-coded.csv
> TYPES=$baseDirHDFS/original-coded-type.csv
> STATS=$baseDirHDFS/univariate-summary.csv
>
> 16/02/04 20:35:03 INFO api.DMLScript: BEGIN DML run 02/04/2016 20:35:03
> 16/02/04 20:35:03 INFO api.DMLScript: HADOOP_HOME: null
> 16/02/04 20:35:03 WARN conf.DMLConfig: No default SystemML config file
> (./SystemML-config.xml) found
> 16/02/04 20:35:03 WARN conf.DMLConfig: Using default settings in DMLConfig
> 16/02/04 20:35:04 WARN hops.OptimizerUtils: Auto-disable multi-threaded
> text read for 'text' and 'csv' due to thread contention on JRE < 1.8
> (java.version=1.7.0_71).
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
>
> [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
>
> [jar:file:/usr/local/explorys/datagrid/lib/slf4j-jdk14-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
>
> [jar:file:/usr/local/explorys/datagrid/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> 16/02/04 20:35:07 INFO api.DMLScript: SystemML Statistics:
> Total execution time:        0.880 sec.
> Number of executed MR Jobs:    0.
>
> 16/02/04 20:35:07 INFO api.DMLScript: END DML run 02/04/2016 20:35:07
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.apache.hadoop.mapred.JobConf.getDouble(Ljava/lang/String;D)D
>     at
>
> org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1195)
>     at
>
> org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1129)
>     at
>
> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:307)
>     at
>
> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:289)
>     at
> org.apache.sysml.runtime.matrix.CSVReblockMR.runJob(CSVReblockMR.java:275)
>     at
> org.apache.sysml.lops.runtime.RunMRJobs.submitJob(RunMRJobs.java:257)
>     at
>
> org.apache.sysml.lops.runtime.RunMRJobs.prepareAndSubmitJob(RunMRJobs.java:143)
>     at
>
> org.apache.sysml.runtime.instructions.MRJobInstruction.processInstruction(MRJobInstruction.java:1500)
>     at
>
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:309)
>     at
>
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:227)
>     at
>
> org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:169)
>     at
> org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:146)
>     at org.apache.sysml.api.DMLScript.execute(DMLScript.java:676)
>     at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:338)
>     at org.apache.sysml.api.DMLScript.main(DMLScript.java:197)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>     at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:606)
>     at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message