systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deron Eriksson <deroneriks...@gmail.com>
Subject Re: Compatibility with MR1 Cloudera cdh4.2.1
Date Fri, 05 Feb 2016 07:04:20 GMT
Hi Ethan,

Just FYI, I looked at hadoop-common-2.0.0-cdh4.2.1.jar (
https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-common/2.0.0-cdh4.2.1/),
since I don't see a 2.0.0-mr1-cdh4.2.1 version, and the
org.apache.hadoop.conf.Configuration class in that jar doesn't appear to
have a getDouble method, so using that version of hadoop-common won't work.

However, the hadoop-common-2.4.1.jar (
https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-common/2.4.1/)
does appear to have the getDouble method. It's possible that adding that
jar to your classpath may fix your problem, as Shirish pointed out.

It sounds like Matthias may have another fix.

Deron



On Thu, Feb 4, 2016 at 6:40 PM, Matthias Boehm <mboehm@us.ibm.com> wrote:

> well, we did indeed not run on MR v1 for a while now. However, I don't
> want to get that far and say we don't support it anymore. I'll fix this
> particular issue by tomorrow.
>
> In the next couple of weeks we should run our full performance testsuite
> (for broad coverage) over an MR v1 cluster and systematically remove
> unnecessary incompatibility like this instance. Any volunteers?
>
> Regards,
> Matthias
>
> [image: Inactive hide details for Ethan Xu ---02/04/2016 05:51:28
> PM---Hello, I got an error when running the systemML/scripts/Univar-S]Ethan
> Xu ---02/04/2016 05:51:28 PM---Hello, I got an error when running the
> systemML/scripts/Univar-Stats.dml script on
>
> From: Ethan Xu <ethan.yifanxu@gmail.com>
> To: dev@systemml.incubator.apache.org
> Date: 02/04/2016 05:51 PM
> Subject: Compatibility with MR1 Cloudera cdh4.2.1
> ------------------------------
>
>
>
> Hello,
>
> I got an error when running the systemML/scripts/Univar-Stats.dml script on
> a hadoop cluster (Cloudera CDH4.2.1) on a 6GB data set. Error message is at
> the bottom of the email. The same script ran fine on a smaller sample
> (several MB) of the same data set, when MR was not invoked.
>
> The main error was java.lang.NoSuchMethodError:
> org.apache.hadoop.mapred.JobConf.getDouble()
> Digging deeper, it looks like the CDH4.2.1 version of MR indeed didn't have
> the JobConf.getDouble() method.
>
> The hadoop-core jar of CDH4.2.1 can be found here:
>
> https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-core/2.0.0-mr1-cdh4.2.1/
>
> The calling line of SystemML is line 1194 of
>
> https://github.com/apache/incubator-systemml/blob/master/src/main/java/org/apache/sysml/runtime/matrix/mapred/MRJobConfiguration.java
>
> I was wondering, if the finding is accurate, is there a potential fix, or
> does this mean the current version of SystemML is not compatible with
> CDH4.2.1?
>
> Thank you,
>
> Ethan
>
>
> hadoop jar $sysDir/target/SystemML.jar -f
> $sysDir/scripts/algorithms/Univar-Stats.dml -nvargs
> X=$baseDirHDFS/original-coded.csv
> TYPES=$baseDirHDFS/original-coded-type.csv
> STATS=$baseDirHDFS/univariate-summary.csv
>
> 16/02/04 20:35:03 INFO api.DMLScript: BEGIN DML run 02/04/2016 20:35:03
> 16/02/04 20:35:03 INFO api.DMLScript: HADOOP_HOME: null
> 16/02/04 20:35:03 WARN conf.DMLConfig: No default SystemML config file
> (./SystemML-config.xml) found
> 16/02/04 20:35:03 WARN conf.DMLConfig: Using default settings in DMLConfig
> 16/02/04 20:35:04 WARN hops.OptimizerUtils: Auto-disable multi-threaded
> text read for 'text' and 'csv' due to thread contention on JRE < 1.8
> (java.version=1.7.0_71).
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
>
> [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
>
> [jar:file:/usr/local/explorys/datagrid/lib/slf4j-jdk14-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
>
> [jar:file:/usr/local/explorys/datagrid/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> 16/02/04 20:35:07 INFO api.DMLScript: SystemML Statistics:
> Total execution time:        0.880 sec.
> Number of executed MR Jobs:    0.
>
> 16/02/04 20:35:07 INFO api.DMLScript: END DML run 02/04/2016 20:35:07
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.apache.hadoop.mapred.JobConf.getDouble(Ljava/lang/String;D)D
>    at
>
> org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1195)
>    at
>
> org.apache.sysml.runtime.matrix.mapred.MRJobConfiguration.setUpMultipleInputs(MRJobConfiguration.java:1129)
>    at
>
> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:307)
>    at
>
> org.apache.sysml.runtime.matrix.CSVReblockMR.runAssignRowIDMRJob(CSVReblockMR.java:289)
>    at
> org.apache.sysml.runtime.matrix.CSVReblockMR.runJob(CSVReblockMR.java:275)
>    at org.apache.sysml.lops.runtime.RunMRJobs.submitJob(RunMRJobs.java:257)
>    at
>
> org.apache.sysml.lops.runtime.RunMRJobs.prepareAndSubmitJob(RunMRJobs.java:143)
>    at
>
> org.apache.sysml.runtime.instructions.MRJobInstruction.processInstruction(MRJobInstruction.java:1500)
>    at
>
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:309)
>    at
>
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:227)
>    at
>
> org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:169)
>    at
> org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:146)
>    at org.apache.sysml.api.DMLScript.execute(DMLScript.java:676)
>    at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:338)
>    at org.apache.sysml.api.DMLScript.main(DMLScript.java:197)
>    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>    at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>    at java.lang.reflect.Method.invoke(Method.java:606)
>    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message