hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yuzhang Han <yuzhanghan1...@gmail.com>
Subject Re: How is the memory usage of containers controlled?
Date Mon, 24 Jun 2013 03:27:47 GMT
Arun, thank you for the tips.
I viewed LinuxResourceCalculatorPlugin. I think it calculates the 
resource usage of a node, since it parses the log files under /proc/. 
But on a single node there can be multiple containers running.
Can you point out in which classes the resource usage of a single 
container is monitored or controlled? Thank you very much!

On 6/21/2013 4:50 PM, Arun C Murthy wrote:
> Start with LinuxResourceCalculatorPlugin.
> Arun
> On Jun 21, 2013, at 1:30 PM, Yuzhang Han <yuzhanghan1982@gmail.com> wrote:
>> Thank you Arun.
>> I am trying to study how YARN works. Can someone tell me which class(es) monitors
containers w.r.t. memory usage?
>> On 6/18/2013 4:07 PM, Arun C Murthy wrote:
>>> NodeManagers monitor containers w.r.t memory usage, and put containers in cgroups
with cpu limits to restrict CPU usage.
>>> On Jun 18, 2013, at 12:30 PM, Yuzhang Han <yuzhanghan1982@gmail.com> wrote:
>>>> Hi,
>>>> I am curious about how YARN containers control their memory usage. Say, I
have a MR job, and I configure that every map task should be assigned a 1 GB container, and
every reduce task a 1.5 GB one. So, when YARN runs the containers, how is it ensured that
all map containers use less than 1 GB memory and all reduce containers less than 1.5 GB? Is
container memory control realized via -Xmx arguments in the scripts generated by YARN to invoke
the MR program in the containers?
>>>> Thank you.
>>> --
>>> Arun C. Murthy
>>> Hortonworks Inc.
>>> http://hortonworks.com/
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/

View raw message