Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-user@hadoop.apache.org
Message-ID: <4F4E937B.7040403@apache.org>
Date: Wed, 29 Feb 2012 13:07:07 -0800
From: Mingjie Lai <mlai@apache.org>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:9.0) Gecko/20111229 Thunderbird/9.0
MIME-Version: 1.0
To: mapreduce-user@hadoop.apache.org
Subject: yarn NoClassDefFoundError from LinuxContainerExecutor
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit


Hi.

I'm trying yarn + security but still cannot make a mapred example 
runing. Can anyone help me to take a look?

My env:
- 3-slave cluster on ec2. Centos 5.5
- nn, dn, rm, nm all started, with security enabled.
- i saw java.lang.NoClassDefFoundError from LinuxContainerExecutor eror 
log: 
./application_1330545370212_0004/container_1330545370212_0004_01_000001/stderr
- If i disable security, still saw this issue.

Any hint?

I followed the instructions from 
http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/ClusterSetup.html

Steps:
I started a mapred sample from nn/rm:

$ usr/lib/hadoop/bin/yarn --config ./conf jar 
share/hadoop/mapreduce/hadoop-mapreduce-examples-0.24.0-SNAPSHOT.jar 
randomwriter 10 10

Logs are from nn, nm,
------------------------------------------
[yarn@ip-10-176-231-35 hadoop]$ /usr/lib/hadoop/bin/yarn --config ./conf 
jar share/hadoop/mapreduce/hadoop-mapreduce-examples-0.24.0-SNAPSHOT.jar 
randomwriter 10 10
Running 30 maps.
Job started: Wed Feb 29 20:33:48 UTC 2012
12/02/29 20:33:48 WARN conf.Configuration: 
mapred.used.genericoptionsparser is deprecated. Instead, use 
mapreduce.client.genericoptionsparser.used
12/02/29 20:33:49 INFO mapreduce.JobSubmitter: number of splits:30
12/02/29 20:33:49 INFO mapred.ResourceMgrDelegate: Submitted application 
application_1330545370212_0005 to ResourceManager at 
ip-10-176-231-35.us-west-1.compute.internal/10.176.231.35:7090
12/02/29 20:33:49 INFO mapreduce.Job: The url to track the job: 
http://ip-10-176-231-35.us-west-1.compute.internal:7050/proxy/application_1330545370212_0005/
12/02/29 20:33:49 INFO mapreduce.Job: Running job: job_1330545370212_0005
12/02/29 20:33:53 INFO mapreduce.Job: Job job_1330545370212_0005 running 
in uber mode : false
12/02/29 20:33:53 INFO mapreduce.Job:  map 0% reduce 0%
12/02/29 20:33:53 INFO mapreduce.Job: Job job_1330545370212_0005 failed 
with state FAILED due to: Application application_1330545370212_0005 
failed 1 times due to AM Container for 
appattempt_1330545370212_0005_000001 exited with  exitCode: 1 due to: 
Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:261)
	at org.apache.hadoop.util.Shell.run(Shell.java:188)
	at 
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
	at 
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:207)
	at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:241)
	at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:68)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

main : command provided 1
main : user is yarn

.Failing this attempt.. Failing the application.
12/02/29 20:33:53 INFO mapreduce.Job: Counters: 0
Job ended: Wed Feb 29 20:33:53 UTC 2012
The job took 5 seconds.
------------------------------------------

LinuxContainer error:

[root@ip-10-176-203-45 yarn]# more 
./application_1330545370212_0004/container_1330545370212_0004_01_000001/stderr
Exception in thread "main" java.lang.NoClassDefFoundError: 
org/apache/hadoop/yar
n/service/CompositeService
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:14
1)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
	at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Caused by: java.lang.ClassNotFoundException: 
org.apache.hadoop.yarn.service.Comp
ositeService
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
	... 12 more
Could not find the main class: 
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.
Program will exit.
----------------------------------------