hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Trevor Robinson <tre...@scurrilous.com>
Subject Re: TestDFSIO job hangs (0.23.1-cdh4b2)
Date Mon, 14 May 2012 22:40:24 GMT
I figured out why this was hung in the single node case: The node
manager was failing to start because the resource manager was already
listening on the same port. I had followed Cloudera's example YARN
setup, which incorrectly (or at least unwisely) uses port 8040 for
yarn.resourcemanager.address:

https://ccp.cloudera.com/display/CDH4B2/Deploying+MapReduce+v2+%28YARN%29+on+a+Cluster

Switching to the default port (8032), both single and clustered
configurations now fail due to this:

Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/mapreduce/v2/app/MRAppMaster
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.mapreduce.v2.app.MRAppMaster
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class:
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.  Program will exit.

How can I fix the classpath to include the appropriate JAR
(hadoop-mapreduce-client-app-0.23.1-cdh4.0.0b2.jar)? It does appear to
be on the nodemanager classpath.

Thanks,
Trevor

On Mon, May 14, 2012 at 3:46 PM, Trevor Robinson <trevor@scurrilous.com> wrote:
> Would someone please give me some troubleshooting tips for TestDFSIO
> hanging on a new 0.23.1-cdh4b2 cluster? I've tried both a 5-machine
> cluster and just running everything on a single node. It's my first
> time configuring YARN, so maybe I've misconfigured something. I don't
> see anything suspicious in the logs for namenode, datanode,
> resourcemanager, or nodemanager.
>
> $ sudo su hdfs -c 'bin/hadoop --config etc/hadoop jar
> ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-0.23.1-cdh4.0.0b2-tests.jar
> TestDFSIO -write -nrFiles 10 -fileSize 1GB'
> 12/05/14 15:02:11 INFO fs.TestDFSIO: TestDFSIO.0.0.6
> 12/05/14 15:02:11 INFO fs.TestDFSIO: nrFiles = 10
> 12/05/14 15:02:11 INFO fs.TestDFSIO: fileSize (MB) = 1024.0
> 12/05/14 15:02:11 INFO fs.TestDFSIO: bufferSize = 1000000
> 12/05/14 15:02:11 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
> 12/05/14 15:02:11 INFO fs.TestDFSIO: creating control file: 1073741824
> bytes, 10 files
> 12/05/14 15:02:12 INFO fs.TestDFSIO: created control files for: 10 files
> 12/05/14 15:02:13 INFO mapred.FileInputFormat: Total input paths to process : 10
> 12/05/14 15:02:13 INFO mapreduce.JobSubmitter: number of splits:10
> 12/05/14 15:02:13 WARN conf.Configuration: mapred.jar is deprecated.
> Instead, use mapreduce.job.jar
> 12/05/14 15:02:13 WARN conf.Configuration: mapred.reduce.tasks is
> deprecated. Instead, use mapreduce.job.reduces
> 12/05/14 15:02:13 WARN conf.Configuration: mapred.output.value.class
> is deprecated. Instead, use mapreduce.job.output.value.class
> 12/05/14 15:02:13 WARN conf.Configuration:
> mapred.used.genericoptionsparser is deprecated. Instead, use
> mapreduce.client.genericoptionsparser.used
> 12/05/14 15:02:13 WARN conf.Configuration: mapred.job.name is
> deprecated. Instead, use mapreduce.job.name
> 12/05/14 15:02:13 WARN conf.Configuration: mapred.input.dir is
> deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
> 12/05/14 15:02:13 WARN conf.Configuration: mapred.output.dir is
> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
> 12/05/14 15:02:13 WARN conf.Configuration: mapred.map.tasks is
> deprecated. Instead, use mapreduce.job.maps
> 12/05/14 15:02:13 WARN conf.Configuration: mapred.output.key.class is
> deprecated. Instead, use mapreduce.job.output.key.class
> 12/05/14 15:02:13 WARN conf.Configuration: io.bytes.per.checksum is
> deprecated. Instead, use dfs.bytes-per-checksum
> 12/05/14 15:02:13 WARN conf.Configuration: mapred.working.dir is
> deprecated. Instead, use mapreduce.job.working.dir
> 12/05/14 15:02:13 INFO mapred.ResourceMgrDelegate: Submitted
> application application_1337025701572_0001 to ResourceManager at
> server1/10.10.130.30:8040
> 12/05/14 15:02:13 INFO mapreduce.Job: The url to track the job:
> http://server1:8088/proxy/application_1337025701572_0001/
> 12/05/14 15:02:13 INFO mapreduce.Job: Running job: job_1337025701572_0001
> <30 minutes pass - no significant CPU or disk activity>
>
> Thanks,
> Trevor

Mime
View raw message