hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rohith Sharma K S <rohithsharm...@huawei.com>
Subject RE: mapred job pending at "Starting scan to move intermediate done files"
Date Thu, 23 Oct 2014 06:18:16 GMT
Hi,

This is problem with your memory configurations in cluster. You have configured "yarn.nodemanager.resource.memory-mb"
as 64MB which is too low.


1.       ApplicationMaster required 2GB to launch container but cluster memory it self has
64MB. So container never get assigned.

2.       Further steps, map memory is 64MB, but map.opts has 1024MB in mapred-site.xml. Again
it is contradictory.

Change NodeManger memory to 8GB and map/reduce memory to 2GB. Try running job.

Thanks & Regards
Rohith Sharma K S

This e-mail and its attachments contain confidential information from HUAWEI, which is intended
only for the person or entity whose address is listed above. Any use of the information contained
herein in any way (including, but not limited to, total or partial disclosure, reproduction,
or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive
this e-mail in error, please notify the sender by phone or email immediately and delete it!

From: mail list [mailto:louis.hust.ml@gmail.com]
Sent: 23 October 2014 07:55
To: user@hadoop.apache.org
Subject: mapred job pending at "Starting scan to move intermediate done files"

hi, all,

I am new to hadoop, and I install the hadoop-2.5.1 on ubuntu  with Pseudo-distributed mode.
When I run a mapped job, the job output the following logs:

louis@ubuntu:~/src/hadoop-book$ hadoop jar hadoop-examples.jar v3.MaxTemperatureDriver input/ncdc/all
max-temp
14/10/22 19:09:56 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/10/22 19:09:57 INFO input.FileInputFormat: Total input paths to process : 2
14/10/22 19:09:58 INFO mapreduce.JobSubmitter: number of splits:2
14/10/22 19:09:58 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1414030015373_0001
14/10/22 19:09:58 INFO impl.YarnClientImpl: Submitted application application_1414030015373_0001
14/10/22 19:09:58 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1414030015373_0001/
14/10/22 19:09:58 INFO mapreduce.Job: Running job: job_1414030015373_0001

As you see, the job halt. Then I check the jps output:

louis@ubuntu:~/src/hadoop-2.5.1$<mailto:louis@ubuntu:~/src/hadoop-2.5.1$> jps
22433 SecondaryNameNode
22716 NodeManager
22240 DataNode
22577 ResourceManager
23083 JobHistoryServer
23148 Jps
22080 NameNode

It seems nothing wrong, then i check the  mapred-louis-historyserver-ubuntu.log:

2014-10-22 19:09:03,831 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner
started
2014-10-22 19:09:03,837 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner
complete
2014-10-22 19:11:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to
move intermediate done files
2014-10-22 19:14:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to
move intermediate done files
2014-10-22 19:17:33,832 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to
move intermediate done files
2014-10-22 19:20:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to
move intermediate done files

Then i check the web ui:
It seems the job is pending!!!!

The attachment contains some configuration files at etc/hadoop/ .
Any idea will be appreciated !





Mime
View raw message