Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of wheeleast@gmail.com designates
 209.85.212.170 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAAqB__uKA155=9z=arPTUMSO_vabnXoGMBOJ8_V5NsN6QcPxbg@mail.gmail.com>
References: 
 <CAAqB__uKA155=9z=arPTUMSO_vabnXoGMBOJ8_V5NsN6QcPxbg@mail.gmail.com>
From: Wangda Tan <wheeleast@gmail.com>
Date: Thu, 21 Aug 2014 21:31:06 +0800
Message-ID: 
 <CAD++eCk=MPat7yWBRDJJjQJu8du6S+jVTpp7eMCikAhdydysFA@mail.gmail.com>
Subject: Re: Memory settings in hadoop YARN
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=047d7b3a89626339f4050123bd58

--047d7b3a89626339f4050123bd58
Content-Type: text/plain; charset=UTF-8

Hi Narayanan,
I've read a great blog post by Rohit Bakhshi before, recommend it to you :
http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/. I
think most your questions are covered by this blog post. Please let me know
if you have more questions.

Thanks,
Wangda Tan


On Wed, Aug 20, 2014 at 12:12 PM, Narayanan K <knarayanan88@gmail.com>
wrote:

> Hi
>
> We run our Pig jobs in Hadoop 0.23 which has the new YARN architecture.
>
> I had few questions on memory used by the jobs :
>
> We have following settings for memory.
>
> mapred.child.java.opts
>
> mapreduce.map.memory.mb
>
> mapreduce.reduce.memory.mb
>
> yarn.app.mapreduce.am.resource.mb
>
> yarn.app.mapreduce.am.command-opts
>
>
> 1. I want to understand these settings to make better use of Hadoop
> cluster.
> 2. How is Memory Allocated to Container ? Does any of the above
> settings result in change in container ?
> 3. Any other memory settings we need to be aware of ?
> 4. I heard there was a virtual memory and physical memory involved. Is
> there any propery documentation/guide that can make the memory
> management easy.
>
> Thanks
> Narayanan
>

--047d7b3a89626339f4050123bd58
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Narayanan,<div>I&#39;ve read a great blog post by Rohit=
 Bakhshi before, recommend it to you :=C2=A0<a href=3D"http://hortonworks.c=
om/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/">http://hortonworks.com/=
blog/how-to-plan-and-configure-yarn-in-hdp-2-0/</a>. I think most your ques=
tions are covered by this blog post. Please let me know if you have more qu=
estions.=C2=A0</div>

<div><br></div><div>Thanks,</div><div>Wangda Tan</div><div><br></div></div>=
<div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Wed, Aug 2=
0, 2014 at 12:12 PM, Narayanan K <span dir=3D"ltr">&lt;<a href=3D"mailto:kn=
arayanan88@gmail.com" target=3D"_blank">knarayanan88@gmail.com</a>&gt;</spa=
n> wrote:<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">Hi<br>
<br>
We run our Pig jobs in Hadoop 0.23 which has the new YARN architecture.<br>
<br>
I had few questions on memory used by the jobs :<br>
<br>
We have following settings for memory.<br>
<br>
mapred.child.java.opts<br>
<br>
mapreduce.map.memory.mb<br>
<br>
mapreduce.reduce.memory.mb<br>
<br>
yarn.app.mapreduce.am.resource.mb<br>
<br>
yarn.app.mapreduce.am.command-opts<br>
<br>
<br>
1. I want to understand these settings to make better use of Hadoop cluster=
.<br>
2. How is Memory Allocated to Container ? Does any of the above<br>
settings result in change in container ?<br>
3. Any other memory settings we need to be aware of ?<br>
4. I heard there was a virtual memory and physical memory involved. Is<br>
there any propery documentation/guide that can make the memory<br>
management easy.<br>
<br>
Thanks<br>
<span class=3D"HOEnZb"><font color=3D"#888888">Narayanan<br>
</font></span></blockquote></div><br></div>

--047d7b3a89626339f4050123bd58--