hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Guttadauro, Jeff" <jeff.guttada...@here.com>
Subject RE: Hadoop 2.7.2 Yarn Memory Utiliziation
Date Fri, 07 Oct 2016 20:32:04 GMT
I encountered under-utilization while moving in particular one of my MR jobs to YARN/Hadoop
2.7 a while back.  While it will probably depend on what your job is doing, in my case, I
think the biggest improvement came for me when I increased the split size (from its default
of 128M up to 2G) since it seemed like my mappers were finishing very quickly, so my containers
were very short-lived, and I was suffering from the impact of allocation/deallocation of container
resources.  I also bumped up memory on the resource manager and did a couple other things.

Here’s the thread:

Hope this helps!  Good luck!

From: George Liaw [mailto:george.a.liaw@gmail.com]
Sent: Friday, October 07, 2016 2:32 PM
To: user@hadoop.apache.org
Subject: Hadoop 2.7.2 Yarn Memory Utiliziation

I'm playing around with setting up a Hadoop 2.7.2 cluster with some configs that I inherited
and I'm running into an issue where YARN isn't utilizing all of the memory available on the
nodemanagers and seems to be limited to 2-3 map tasks for an application for some reason.
Can anyone shed some light on this or let me know what else I should look into?

Attached screenshots below of what I'm seeing.

Relevant configs that I'm aware of:




[Inline image 5][Inline image 4][Inline image 6]


George A. Liaw

View raw message