hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Guttadauro, Jeff" <jeff.guttada...@here.com>
Subject RE: Hadoop 2.7.2 Yarn Memory Utiliziation
Date Fri, 07 Oct 2016 20:32:04 GMT
I encountered under-utilization while moving in particular one of my MR jobs to YARN/Hadoop
2.7 a while back.  While it will probably depend on what your job is doing, in my case, I
think the biggest improvement came for me when I increased the split size (from its default
of 128M up to 2G) since it seemed like my mappers were finishing very quickly, so my containers
were very short-lived, and I was suffering from the impact of allocation/deallocation of container
resources.  I also bumped up memory on the resource manager and did a couple other things.

Here’s the thread:
https://mail-archives.apache.org/mod_mbox/hadoop-user/201605.mbox/%3CAM2PR04MB07084248711DC9138D805A9990420@AM2PR04MB0708.eurprd04.prod.outlook.com%3E

Hope this helps!  Good luck!
-Jeff

From: George Liaw [mailto:george.a.liaw@gmail.com]
Sent: Friday, October 07, 2016 2:32 PM
To: user@hadoop.apache.org
Subject: Hadoop 2.7.2 Yarn Memory Utiliziation

Hi,
I'm playing around with setting up a Hadoop 2.7.2 cluster with some configs that I inherited
and I'm running into an issue where YARN isn't utilizing all of the memory available on the
nodemanagers and seems to be limited to 2-3 map tasks for an application for some reason.
Can anyone shed some light on this or let me know what else I should look into?

Attached screenshots below of what I'm seeing.

Relevant configs that I'm aware of:
yarn-site.xml:

<property>
  <name>yarn.scheduler.minimum-allocation-mb</name>
  <value>128</value>
</property>

<property>
  <name>yarn.scheduler.maximum-allocation-mb</name>
  <value>4096</value>
</property>
mapred-site.xml:

<property>
  <name>mapreduce.map.memory.mb</name>
  <value>4096</value>
</property>

<property>
  <name>mapreduce.reduce.memory.mb</name>
  <value>4096</value>
</property>
[Inline image 5][Inline image 4][Inline image 6]

Thanks,
George

--
George A. Liaw

Mime
View raw message