hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grandl Robert <rgra...@yahoo.com.INVALID>
Subject Re: crossPlatformifyMREnv exception
Date Thu, 19 Jun 2014 21:59:27 GMT
Hmm, 


Things are really weird. So If I don't add TEZ jars/conf in hadoop_classpath simply running
Mapreduce jobs works. If I add TEZ jars/conf to hadoop_classpath and try to run mapreduce
jobs, it fails with the exception:
14/06/19 14:46:40 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/hadoop/.staging/job_1403214320220_0001
java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.v2.util.MRApps.crossPlatformifyMREnv(Lorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/yarn/api/ApplicationConstants$Environment;)Ljava/lang/String;
    at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:390)
    at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:284)
    


If I add TEZ jars/conf to hadoop classpath BUT also change framework name from yarn to yarn-tez
and run tez application(like orderedwordcount) it succeeds. Now I can also run hadoop mapreduce
apps, but still as tez. I am completely confused why I cannot run just mapreduce jobs(with
framework name = yarn) when adding TEZ in classpath. Is this a bug ?

robert




On Thursday, June 19, 2014 2:34 PM, Jian He <jhe@hortonworks.com> wrote:
 


The problem looks like it's referencing an old jar which doesn't have this
method. are you running on single node cluster? can you check you cluster
setting about jar dependency. If it's easy for you, just refresh the
environment and re-deploy the single node cluster.

Jian


On Thu, Jun 19, 2014 at 9:15 AM, Grandl Robert <rgrandl@yahoo.com.invalid>
wrote:

> Any suggestion related to this ?
>
> Thanks,
> robert
>
>
>
> On Wednesday, June 18, 2014 11:56 PM, Grandl Robert
> <rgrandl@yahoo.com.INVALID> wrote:
>
>
>
> I am using 2.4 for client as well. Actually I took a tar gz from
> http://apache.petsads.us/hadoop/common/hadoop-2.4.0/,
>
> and I am trying to run. I tried even one node. I am lack of ideas why this
> happens.
>
>
>
>
>
> On Wednesday, June 18, 2014 11:52 PM, Jian He <jhe@hortonworks.com> wrote:
>
>
>
> This new method crossPlatformifyMREnv  is newly added in 2.4.0 release,
> which version of MR client are you using?
> can you make sure you have the same version of client jars
>
> Jian
>
>
>
> On Wed, Jun 18, 2014 at 10:37 PM, Grandl Robert <rgrandl@yahoo.com.invalid>
> wrote:
>
> Hi guys,
> >
> >I don't know what I did but my hadoop yarn went crazy. I am not able to
> submit any job, as it throws the following exception.
> >
> >4/06/18 22:25:19 INFO mapreduce.JobSubmitter: number of splits:1
> >14/06/18 22:25:19 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_1403155404621_0001
> >14/06/18 22:25:19 INFO mapreduce.JobSubmitter: Cleaning up the staging
> area /tmp/hadoop-yarn/staging/hadoop/.staging/job_1403155404621_0001
> >java.lang.NoSuchMethodError:
> org.apache.hadoop.mapreduce.v2.util.MRApps.crossPlatformifyMREnv(Lorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/yarn/api/ApplicationConstants$Environment;)Ljava/lang/String;
> >    at
> org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:390)
> >    at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:284)
> >    at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:430)
> >    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
> >    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
> >    at java.security.AccessController.doPrivileged(Native Method)
> >    at javax.security.auth.Subject.doAs(Subject.java:415)
> >    at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> >    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
> >    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1286)
> >    at
> org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
> >    at
> org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
> >    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> >    at
> org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
> >    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >    at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >    at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >    at java.lang.reflect.Method.invoke(Method.java:601)
> >    at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> >
> >
> >I configured all class path and variables as such:
> >export HADOOP_COMMON_HOME=/home/hadoop/hadoop-2.4.0
> >export HADOOP_HOME=$HADOOP_COMMON_HOME
> >export HADOOP_HDFS_HOME=$HADOOP_COMMON_HOME
> >export HADOOP_MAPRED_HOME=$HADOOP_COMMON_HOME
> >export HADOOP_YARN_HOME=$HADOOP_COMMON_HOME
> >export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_MAPRED_HOME/lib/native/
> >export HADOOP_CONF_DIR=/home/hadoop/rgrandl/conf/
> >export YARN_CONF_DIR=/home/hadoop/rgrandl/conf/
> >export HADOOP_BIN_PATH=$HADOOP_MAPRED_HOME/bin/
> >export HADOOP_SBIN=$HADOOP_MAPRED_HOME/sbin/
> >export HADOOP_LOGS=$HADOOP_HOME/logs
> >export HADOOP_LOG_DIR=$HADOOP_HOME/logs
> >export YARN_LOG_DIR=$HADOOP_HOME/logs
> >
> >export JAVA_HOME=/home/hadoop/rgrandl/java/
> >export HADOOP_USER_CLASSPATH_FIRST=1
> >export YARN_HOME=/home/hadoop/hadoop-2.4.0
> >export TEZ_CONF_DIR=/home/hadoop/rgrandl/conf
> >export TEZ_JARS=/home/hadoop/rgrandl/tez/tez-0.4.0-incubating
> >
> >export HADOOP_PREFIX=$HADOOP_COMMON_HOME
> >
> >export
> HADOOP_CLASSPATH=$HADOOP_HOME:/home/hadoop/rgrandl/tez/tez-0.4.0-incubating/*:/home/hadoop/rgrandl/tez/tez-0.4.0-incubating/lib/*:/home/hadoop/rgrandl/hive:/home/hadoop/rgrandl/conf
> >
> >export
> PATH=$PATH:$HADOOP_BIN_PATH:$HADOOP_SBIN:$YARN_CONF_DIR:$HADOOP_YARN_HOME:$HADOOP_MAPRED_HOME:$HADOOP_HDFS_HOME:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME:$JAVA_HOME/bin/:/home/hadoop/rgrandl/hive/bin
> >
> >
> >Everything seems to be correct, but I cannot understand this error. Is
> something I never encountered before.
> >
> >
> >Do you have any hints on it ?
> >
> >Thanks,
> >robert
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message