hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mail list <louis.hust...@gmail.com>
Subject Re: mapred job pending at "Starting scan to move intermediate done files"
Date Thu, 23 Oct 2014 07:59:09 GMT
Hi,

Thanks Rohith,I just try this job on my virtual machine which does not have so much memory.
Can I modify some configuration options to let it run with fewer memory?
If not, i have to try it with a new environment.

Thank you again!

On Oct 23, 2014, at 14:18, Rohith Sharma K S <rohithsharmaks@huawei.com> wrote:

> Hi,
>  
> This is problem with your memory configurations in cluster. You have configured “yarn.nodemanager.resource.memory-mb”
as 64MB which is too low.
>  
> 1.       ApplicationMaster required 2GB to launch container but cluster memory it self
has 64MB. So container never get assigned.
> 2.       Further steps, map memory is 64MB, but map.opts has 1024MB in mapred-site.xml.
Again it is contradictory.
>  
> Change NodeManger memory to 8GB and map/reduce memory to 2GB. Try running job.
>  
> Thanks & Regards
> Rohith Sharma K S
>  
> This e-mail and its attachments contain confidential information from HUAWEI, which is
intended only for the person or entity whose address is listed above. Any use of the information
contained herein in any way (including, but not limited to, total or partial disclosure, reproduction,
or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive
this e-mail in error, please notify the sender by phone or email immediately and delete it!
>  
> From: mail list [mailto:louis.hust.ml@gmail.com] 
> Sent: 23 October 2014 07:55
> To: user@hadoop.apache.org
> Subject: mapred job pending at "Starting scan to move intermediate done files"
>  
> hi, all,
>  
> I am new to hadoop, and I install the hadoop-2.5.1 on ubuntu  with Pseudo-distributed
mode.  
> When I run a mapped job, the job output the following logs:
>  
> louis@ubuntu:~/src/hadoop-book$ hadoop jar hadoop-examples.jar v3.MaxTemperatureDriver
input/ncdc/all max-temp
> 14/10/22 19:09:56 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
> 14/10/22 19:09:57 INFO input.FileInputFormat: Total input paths to process : 2
> 14/10/22 19:09:58 INFO mapreduce.JobSubmitter: number of splits:2
> 14/10/22 19:09:58 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1414030015373_0001
> 14/10/22 19:09:58 INFO impl.YarnClientImpl: Submitted application application_1414030015373_0001
> 14/10/22 19:09:58 INFO mapreduce.Job: The url to track the job:http://localhost:8088/proxy/application_1414030015373_0001/
> 14/10/22 19:09:58 INFO mapreduce.Job: Running job: job_1414030015373_0001
>  
> As you see, the job halt. Then I check the jps output:
>  
> louis@ubuntu:~/src/hadoop-2.5.1$ jps
> 22433 SecondaryNameNode
> 22716 NodeManager
> 22240 DataNode
> 22577 ResourceManager
> 23083 JobHistoryServer
> 23148 Jps
> 22080 NameNode
>  
> It seems nothing wrong, then i check the  mapred-louis-historyserver-ubuntu.log:
>  
> 2014-10-22 19:09:03,831 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner
started
> 2014-10-22 19:09:03,837 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner
complete
> 2014-10-22 19:11:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan
to move intermediate done files
> 2014-10-22 19:14:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan
to move intermediate done files
> 2014-10-22 19:17:33,832 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan
to move intermediate done files
> 2014-10-22 19:20:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan
to move intermediate done files
>  
> Then i check the web ui:
> It seems the job is pending!!!!
>  
> The attachment contains some configuration files at etc/hadoop/ . 
> Any idea will be appreciated !


Mime
View raw message