hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ray Chiang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6941) The default setting doesn't work for MapReduce job
Date Fri, 25 Aug 2017 22:54:01 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142381#comment-16142381

Ray Chiang commented on MAPREDUCE-6941:

The ClusterSetup and SingleCluster documents were updated to do proper setup and there was
a lot of discussion about the "right" way to deal with this.  It's not obvious that missing
the new setting will cause the above error.  We don't really have any "common error messages
for debugging" documentation.

[~djp], I don't think we'll be able to update the defaults without getting into the same discussions
and I don't have any newer suggestions offhand.

> The default setting doesn't work for MapReduce job
> --------------------------------------------------
>                 Key: MAPREDUCE-6941
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6941
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 3.0.0-beta1
>            Reporter: Junping Du
>            Priority: Blocker
> On the deployment of hadoop 3 cluster (based on current trunk branch) with default settings,
the MR job will get failed as following exceptions:
> {noformat}
> 2017-08-16 13:00:03,846 INFO mapreduce.Job: Job job_1502913552390_0001 running in uber
mode : false
> 2017-08-16 13:00:03,847 INFO mapreduce.Job:  map 0% reduce 0%
> 2017-08-16 13:00:03,864 INFO mapreduce.Job: Job job_1502913552390_0001 failed with state
FAILED due to: Application application_1502913552390_0001 failed 2 times due to AM Container
for appattempt_1502913552390_0001_000002 exited with  exitCode: 1
> Failing this attempt.Diagnostics: [2017-08-16 13:00:02.963]Exception from container-launch.
> Container id: container_1502913552390_0001_02_000001
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> 	at org.apache.hadoop.util.Shell.runCommand(Shell.java:994)
> 	at org.apache.hadoop.util.Shell.run(Shell.java:887)
> 	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1212)
> 	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:295)
> 	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:455)
> 	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:275)
> 	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:90)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This is because mapreduce related jar are not added into yarn setup by default. To make
MR job run successful, we need to add following configurations to yarn-site.xml now:
> {noformat}
>     <property>
>       <name>yarn.application.classpath</name>
>       <value>
>         ...
>         <HADOOP_HOME>/share/hadoop/mapreduce/*,
>         <HADOOP_HOME>/share/hadoop/mapreduce/lib/*
>         ...
>       </value>
> {noformat}
> But this config is not necessary for previous version of Hadoop. We should fix this issue
before beta release otherwise it will be a regression for configuration changes.
> This could be more like a YARN issue (if so, we should move), depends on how we fix it

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message