hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naganarasimha G R (Naga)" <garlanaganarasi...@huawei.com>
Subject RE: Why would ApplicationManager request RAM more that defaut 1GB?
Date Wed, 23 Sep 2015 04:21:54 GMT
Hi Ilya,
    In a normal case AM memory requirement should not be more than the default for small sized
jobs, but seems to be something erroneous in your case, Would like to have more information
:
1. Which version of Hadoop/ YARN ?
2. From the logs is it getting killed due to over usage of Vmem or Physical memory ?
3. Can you run " jmap -histo -F <PID of AM container>" and share the heap dump result?
4. If possible can you pastebin the AM logs?

+ Naga
________________________________________
From: Ilya Karpov [i.karpov@cleverdata.ru]
Sent: Tuesday, September 22, 2015 21:06
To: user@hadoop.apache.org
Subject: Why would ApplicationManager request RAM more that defaut 1GB?

Hi all,
can’t figure out subj.
On my hadoop cluster I have an issue when ApplicationMaster(AM) killed by NodeManager because
AM tries to allocate more than default 1GB. MR application, that AM is in charge of, is a
mapper only job(1(!) mapper, no reducers, downloads data from remote source). At the moment
when AM killed, MR job is ok (uses about 70% of ram limit). MR job doesn't have any custom
counters, distributes caches etc, just downloads data (by portions) via custom input format.
To fix this issue, I raised memory limit for AM, but I want to know what is the reason of
eating 1GB (!) for a trivial job like mine?


Mime
View raw message