hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joseph Naegele" <jnaeg...@grierforensics.com>
Subject YarnChild and Container running beyond physical memory limits
Date Wed, 13 Apr 2016 23:40:30 GMT


Can anyone tell me what exactly YarnChild is and how I can control the
quantity of child JVMs running in each container? In this case I'm concerned
with the map phase of my MR job. I'm having issues with my containers
running beyond *physical* memory limits and I'm trying to determine the


Is each child JVM just an individual map task? If so, why do I see a
variable number of them? I don't know if each of these JVMs is a clone of
the original YarnChild process, what they are doing, why they are each using
so much memory (1G).


Here is a sample excerpt of my MR job when YARN kills a container:

Here's the same process tree reorganized and ordered by ancestry:


If I increase the amount of memory per container, in turn lowering the total
number of containers, I see these errors less often as expected, BUT when I
do see them, there are NO child JVM processes and it's always due to some
other unrelated external process chewing up RAM. Here is an example of that:
7/raw/934a93a7afe09c7cd62a50edc08ce902b9e71aac/job.log. You can see that the
[redacted] process is the culprit in that case.


I can share my mapred/yarn configuration if it's helpful.


If anyone has any ideas I'd greatly appreciate them!




View raw message