hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5831) Old MR client is not compatible with new MR application
Date Fri, 26 Sep 2014 04:21:36 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14148724#comment-14148724
] 

Hadoop QA commented on MAPREDUCE-5831:
--------------------------------------

{color:green}+1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12671374/MAPREDUCE-5831-v3.patch
  against trunk revision e96ce6f.

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 1 new or modified
test files.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

    {color:green}+1 eclipse:eclipse{color}.  The patch built with eclipse:eclipse.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version
2.0.3) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:green}+1 core tests{color}.  The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

    {color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4917//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4917//console

This message is automatically generated.

> Old MR client is not compatible with new MR application
> -------------------------------------------------------
>
>                 Key: MAPREDUCE-5831
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5831
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: client, mr-am
>    Affects Versions: 2.2.0, 2.3.0
>            Reporter: Zhijie Shen
>            Assignee: Junping Du
>            Priority: Blocker
>         Attachments: MAPREDUCE-5831-v2.patch, MAPREDUCE-5831-v3.patch, MAPREDUCE-5831.patch
>
>
> Recently, we saw the following scenario:
> 1. The user setup a cluster of hadoop 2.3., which contains YARN 2.3 and MR  2.3.
> 2. The user client on a machine that MR 2.2 is installed and in the classpath.
> Then, when the user submitted a simple wordcount job, he saw the following message:
> {code}
> 16:00:41,027  INFO main mapreduce.Job:1345 -  map 100% reduce 100%
> 16:00:41,036  INFO main mapreduce.Job:1356 - Job job_1396468045458_0006 completed successfully
> 16:02:20,535  WARN main mapreduce.JobRunner:212 - Cannot start job [wordcountJob]
> java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.mapreduce.JobCounter.MB_MILLIS_REDUCES
> 	at java.lang.Enum.valueOf(Enum.java:236)
> 	at org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.valueOf(FrameworkCounterGroup.java:148)
> 	at org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.findCounter(FrameworkCounterGroup.java:182)
> 	at org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:154)
> 	at org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:240)
> 	at org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:370)
> 	at org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511)
> 	at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756)
> 	at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> 	at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753)
> 	at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361)
> 	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1289)
>         . . .
> {code}
> The problem is that the wordcount job was running on one or more than one nodes of the
YARN cluster, where MR 2.3 libs were installed, and JobCounter.MB_MILLIS_REDUCES is available
in the counters. On the other side, due to the classpath setting, the client was likely to
run with MR 2.2 libs. After the client retrieved the counters from MR AM, it tried to construct
the Counter object with the received counter name. Unfortunately, the enum didn't exist in
the client's classpath. Therefore, "No enum constant" exception is thrown here.
> JobCounter.MB_MILLIS_REDUCES is brought to MR2 via MAPREDUCE-5464 since Hadoop 2.3.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message