hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grandl Robert <rgra...@yahoo.com.INVALID>
Subject Re: crossPlatformifyMREnv exception
Date Thu, 19 Jun 2014 23:22:26 GMT
Hi Hitesh,

It seems tez is using 2.2 while hadoop-2.4 has 2-4 for mapreduce. .....Is this the problem
?
If yes, how can I fix ?


For tez-0.4 they are:
hadoop@nectar-11:~/rgrandl$ ls tez/tez-0.4.0-incubating/lib/
avro-1.7.4.jar                             hadoop-mapreduce-client-core-2.2.0.jar
commons-cli-1.2.jar                        hadoop-mapreduce-client-shuffle-2.2.0.jar
commons-collections4-4.0.jar               jettison-1.3.4.jar
guava-11.0.2.jar                           protobuf-java-2.5.0.jar
guice-3.0.jar                              snappy-java-1.0.4.1.jar
hadoop-mapreduce-client-common-2.2.0.jar   


For Hadoop-2.4 they are:
hadoop@nectar-11:~/rgrandl$ ls ~/hadoop-2.4.0/share/hadoop/mapreduce/
hadoop-mapreduce-client-app-2.4.0.jar          hadoop-mapreduce-client-jobclient-2.4.0-tests.jar
hadoop-mapreduce-client-common-2.4.0.jar      hadoop-mapreduce-client-shuffle-2.4.0.jar
hadoop-mapreduce-client-core-2.4.0.jar          hadoop-mapreduce-examples-2.4.0.jar
hadoop-mapreduce-client-hs-2.4.0.jar          lib
hadoop-mapreduce-client-hs-plugins-2.4.0.jar  lib-examples
hadoop-mapreduce-client-jobclient-2.4.0.jar   sources


robert



On Thursday, June 19, 2014 4:15 PM, Hitesh Shah <hitesh@apache.org> wrote:
 


Hi Robert, 

If you look at the tez jars, you will probably see a couple of mapreduce jars that Tez depends
on under TEZ_BASE_DIR/lib/. Are these jars inconsistent with respect to the version of hadoop/mapreduce
on the cluster? 

— Hitesh


On Jun 19, 2014, at 2:59 PM, Grandl Robert <rgrandl@yahoo.com.INVALID> wrote:

> Hmm, 
> 
> 
> Things are really weird. So If I don't add TEZ jars/conf in hadoop_classpath simply running
Mapreduce jobs works. If I add TEZ jars/conf to hadoop_classpath and try to run mapreduce
jobs, it fails with the exception:
> 14/06/19 14:46:40 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/hadoop/.staging/job_1403214320220_0001
> java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.v2.util.MRApps.crossPlatformifyMREnv(Lorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/yarn/api/ApplicationConstants$Environment;)Ljava/lang/String;
>     at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:390)
>     at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:284)
>    
> 
> 
> If I add TEZ jars/conf to hadoop classpath BUT also change framework name from yarn to
yarn-tez and run tez application(like orderedwordcount) it succeeds. Now I can also run hadoop
mapreduce apps, but still as tez. I am completely confused why I cannot run just mapreduce
jobs(with framework name = yarn) when adding TEZ in classpath. Is this a bug ?
> 
> robert
> 
> 
> 
> 
> On Thursday, June 19, 2014 2:34 PM, Jian He <jhe@hortonworks.com> wrote:
> 
> 
> 
> The problem looks like it's referencing an old jar which doesn't have this
> method. are you running on single node cluster? can you check you cluster
> setting about jar dependency. If it's easy for you, just refresh the
> environment and re-deploy the single node cluster.
> 
> Jian
> 
> 
> On Thu, Jun 19, 2014 at 9:15 AM, Grandl Robert <rgrandl@yahoo.com.invalid>
> wrote:
> 
>> Any suggestion related to this ?
>> 
>> Thanks,
>> robert
>> 
>> 
>> 
>> On Wednesday, June 18, 2014 11:56 PM, Grandl Robert
>> <rgrandl@yahoo.com.INVALID> wrote:
>> 
>> 
>> 
>> I am using 2.4 for client as well. Actually I took a tar gz from
>> http://apache.petsads.us/hadoop/common/hadoop-2.4.0/,
>> 
>> and I am trying to run. I tried even one node. I am lack of ideas why this
>> happens.
>> 
>> 
>> 
>> 
>> 
>> On Wednesday, June 18, 2014 11:52 PM, Jian He <jhe@hortonworks.com> wrote:
>> 
>> 
>> 
>> This new method crossPlatformifyMREnv  is newly added in 2.4.0 release,
>> which version of MR client are you using?
>> can you make sure you have the same version of client jars
>> 
>> Jian
>> 
>> 
>> 
>> On Wed, Jun 18, 2014 at 10:37 PM, Grandl Robert <rgrandl@yahoo.com.invalid>
>> wrote:
>> 
>> Hi guys,
>>> 
>>> I don't know what I did but my hadoop yarn went crazy. I am not able to
>> submit any job, as it throws the following exception.
>>> 
>>> 4/06/18 22:25:19 INFO mapreduce.JobSubmitter: number of splits:1
>>> 14/06/18 22:25:19 INFO mapreduce.JobSubmitter: Submitting tokens for job:
>> job_1403155404621_0001
>>> 14/06/18 22:25:19 INFO mapreduce.JobSubmitter: Cleaning up the staging
>> area /tmp/hadoop-yarn/staging/hadoop/.staging/job_1403155404621_0001
>>> java.lang.NoSuchMethodError:
>> org.apache.hadoop.mapreduce.v2.util.MRApps.crossPlatformifyMREnv(Lorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/yarn/api/ApplicationConstants$Environment;)Ljava/lang/String;
>>>     at
>> org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:390)
>>>     at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:284)
>>>     at
>> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:430)
>>>     at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>>>     at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>>>     at java.security.AccessController.doPrivileged(Native Method)
>>>     at javax.security.auth.Subject.doAs(Subject.java:415)
>>>     at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>>>     at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>>>     at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1286)
>>>     at
>> org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
>>>     at
>> org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
>>>     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>     at
>> org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>     at java.lang.reflect.Method.invoke(Method.java:601)
>>>     at
>> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
>>> 
>>> 
>>> I configured all class path and variables as such:
>>> export HADOOP_COMMON_HOME=/home/hadoop/hadoop-2.4.0
>>> export HADOOP_HOME=$HADOOP_COMMON_HOME
>>> export HADOOP_HDFS_HOME=$HADOOP_COMMON_HOME
>>> export HADOOP_MAPRED_HOME=$HADOOP_COMMON_HOME
>>> export HADOOP_YARN_HOME=$HADOOP_COMMON_HOME
>>> export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_MAPRED_HOME/lib/native/
>>> export HADOOP_CONF_DIR=/home/hadoop/rgrandl/conf/
>>> export YARN_CONF_DIR=/home/hadoop/rgrandl/conf/
>>> export HADOOP_BIN_PATH=$HADOOP_MAPRED_HOME/bin/
>>> export HADOOP_SBIN=$HADOOP_MAPRED_HOME/sbin/
>>> export HADOOP_LOGS=$HADOOP_HOME/logs
>>> export HADOOP_LOG_DIR=$HADOOP_HOME/logs
>>> export YARN_LOG_DIR=$HADOOP_HOME/logs
>>> 
>>> export JAVA_HOME=/home/hadoop/rgrandl/java/
>>> export HADOOP_USER_CLASSPATH_FIRST=1
>>> export YARN_HOME=/home/hadoop/hadoop-2.4.0
>>> export TEZ_CONF_DIR=/home/hadoop/rgrandl/conf
>>> export TEZ_JARS=/home/hadoop/rgrandl/tez/tez-0.4.0-incubating
>>> 
>>> export HADOOP_PREFIX=$HADOOP_COMMON_HOME
>>> 
>>> export
>> HADOOP_CLASSPATH=$HADOOP_HOME:/home/hadoop/rgrandl/tez/tez-0.4.0-incubating/*:/home/hadoop/rgrandl/tez/tez-0.4.0-incubating/lib/*:/home/hadoop/rgrandl/hive:/home/hadoop/rgrandl/conf
>>> 
>>> export
>> PATH=$PATH:$HADOOP_BIN_PATH:$HADOOP_SBIN:$YARN_CONF_DIR:$HADOOP_YARN_HOME:$HADOOP_MAPRED_HOME:$HADOOP_HDFS_HOME:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME:$JAVA_HOME/bin/:/home/hadoop/rgrandl/hive/bin
>>> 
>>> 
>>> Everything seems to be correct, but I cannot understand this error. Is
>> something I never encountered before.
>>> 
>>> 
>>> Do you have any hints on it ?
>>> 
>>> Thanks,
>>> robert
>> 
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
> 
>> 
> 
> -- 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to 
> which it is addressed and may contain information that is confidential, 
> privileged and exempt from disclosure under applicable law. If the reader 
> of this message is not the intended recipient, you are hereby notified that 
> any printing, copying, dissemination, distribution, disclosure or 
> forwarding of this communication is strictly prohibited. If you have 
> received this communication in error, please contact the sender immediately 
> and delete it from your system. Thank You.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message